orgtheory.net

An Elaboration on Brayden King’s reaction to Neal Caren’s list of most cited works in sociology

Several people have pointed out Neal Caren lists of most cited works. I appreciate how hard it is to do something like this and I appreciate the work Neal Caren has done. So my criticism is intended more to get us closer to the truth here and to caution against this list getting reified. I also have some suggestions for Neal Caren’s next foray here.

The idea, as I understand it, is to try and create a list of the 100 most cited sociology books and papers in the period 2005-2010. Leaving aside the fact that the SSCI under counts sociology cites by a wide margin, (maybe a factor of 400-500% if you believe what comes out of Google Scholar), the basic problem with the list is that it is not based on a good sample of all of the works in sociology. Because the journals were chosen on an ad hoc basis, one has no idea as to what the bias is in making that choice. The theory Neal Caren is working with, is that these journals are somehow a sample of all sociology journals and that their citation patterns reflect the discipline at large. The only way to make this kind of assertion is to randomly sample from all sociology journals.

The idea here is that if Bourdieu’s Distinctions is really the most cited work in sociology (an inference people are drawing from the list), then it should be equally likely to appear in all sociology articles and all sociology journals at a similar rate. The only way to know if this is true, is to sample all journals or all articles, not some subset chosen purposively. Adding ASQ to this, does not matter because it only adds one more arbitrary choice in a nonrandom sampling scheme. .

I note that the Social Science Citation Index follows 139 Sociology journals. A random sample of 20% would yield 28 journals and looking at those papers across a random sample of journals is going to get us a better idea at finding out which works are the most cited in sociology.

Is there any evidence that the nonrandom sample chosen by Neal Caren is nonrandom? The last three cites on his list include one by Latour (49 cites), Byrk (49 cites) and Blair Loy (49 cites). If one goes to the SSCI and looks up all of the cites to these works  from 2005-2010, not just the ones that appear in these journals, one comes to a startling result: Latour has 1266 cites, Bryk, 124, and Blair Loy 152. At the top of the list, Bourdieu’s Distinctions has 218 on Neal Caren’s list but the SSCI shows Distinctions as having 865 cites overall.  Latour’s book should put him at the top of the list, but the way the journals are chosen here puts him at the bottom. It ought to make anyone who looks at this list nervous, that Latour’s real citation count is 25 times larger than reported and it puts him ahead of Bourdieu’s Distinctions.

The list is also clearly nonrandom for what is left off. Brayden King mentioned that the list is light on organizational and economic sociology. So, I did some checking. Woody Powell’s 1990 paper called “Neither markets nor hierarchies” has 464 cites from 2005-2010 and his paper with three other colleagues that appeared in the AJS in 2005, “Networks dynamics and field evolution” has 267 cites. In my own work, my 1996 ASR paper “Markets as politics” has 363 cites and my 2001 book “The Architecture of Markets” has 454 from 2005-2010. If without much work, I can find four articles or books that have more cites than two of the three bottom cites on the list (i.e. Byrk’s 124 and Blair Loy’s 152 done the same way), there must be lots more missing.

This suggests that if we really want to understand what are the most cited and core works in sociology in any time period,  we cannot use purposive samples of journals. What is required is a substantial number of journals being sampled, and then all of the cites to the papers or books tallied for those books and papers from the SSCI in order to see which works really are the most cited. I assume that many of the books and papers on the list will still be there, i.e. things like Bourdieu, Granovetter, DiMaggio and Powell, Meyer and Rowan, Swidler, and Sewell. But because of the nonrandom sampling, lots of things that appear to be missing are probably, well, missing.

Written by fligstein

June 5, 2012 at 9:32 pm

15 Responses

Subscribe to comments with RSS.

  1. […] glad to have him join us. Neil was so excited to start blogging that he beat us to the punch and already has a post up. Welcome Neil! Share this:TwitterStumbleUponDiggEmailRedditFacebookPrintLinkedInLike this:LikeBe […]

    Like

  2. But doesn’t this assume that the SSCI’s categorization of journals, and hence a random sample of it, is somehow better — closer to “the truth” — than an insider’s categorization of journals? A quick perusal of the SSCI’s list doesn’t give me much confidence in the assumption: among the ten “sociology” journals on the first page are Agriculture and Human Values, Annals of Tourism Research, and Anthrozoos, three journals that most American sociologists, at least, aren’t going to claim as core to the discipline.

    More generally, this strikes me as a classic boundary / definition-of-the-risk-set problem. As we saw with ASQ.

    “The idea here is that if Bourdieu’s Distinctions is really the most cited work in sociology (an inference people are drawing from the list), then it should be equally likely to appear in all sociology articles and all sociology journals at a similar rate.”

    I couldn’t really tell if this is Neil’s idea or Neal’s idea (or Neil’s interpretation of Neal’s idea), but in any event it doesn’t make sense. The “then” doesn’t follow from the “if”.

    Like

    krippendorf

    June 5, 2012 at 10:17 pm

  3. I’m generally happy to take credit for Brayden’s ideas, as he has a lot of good ones, but I should probably say that it was he, and not I, who noted Caren’s list was a bit light on Econ Soc.

    And welcome, Neil—it’s just terrific to have you as a guest.

    Like

    Kieran

    June 5, 2012 at 11:50 pm

  4. I too think Neal’s list is better than SSCI’s. Isn’t there some systematic way of defining a soc journal, such as by the preponderance of degrees of the people who publish in it, or their affiliations?

    Like

    Philip N. Cohen

    June 6, 2012 at 12:34 am

  5. Just changed it.

    I will write something about the “Theory of Fields” soon.

    Like

    fligstein

    June 6, 2012 at 12:48 am

  6. Who says stuff on the internet isn’t peer reviewed?

    If anyone is interested in the background of the project, you might want to check out the first version from about two weeks ago. It looks at just ASR, AJS, Social Forces and Social Problems. I also include the Python code, so anyone who wanted could insert journal data and produce their own analysis.

    As I noted at the time, I didn’t like relying on this list of journals. Mostly, I didn’t like that it excluded books. But I also was bothered that it included only the elite journals. So I went to Web of Science. They list about 18,000 articles published in the last five years in their Sociology category, which is actually a manageable number. But a quick look at the list let me to believe gathering them all was including many things that most sociologists would not considered primarily sociology journals, such as Human Ecology, Cornell Hospital Quarterly, Human Studies. It also excluded journals that many would argue are actually sociology journals, like Administrative Science Quarterly or Demography. So based on the Web of Science list and my understanding of what sociologists I’ve encountered consider sociology journals, I expanded the list. I wasn’t entirely happy with it, but the final list seemed better than either restricting it to the top four journals or using the Web of Science list. To cover all my bases, I specifically requested that if anyone wanted new journals included, they could contact me. To date, two-thirds of the responses to this have been orgtheory posts. What is a sociology journal is fuzzy, but if some selection committee wants to make a list, I’m happy to run the analysis for it.

    Additionally, I posted the top top 10 works that were cited in each of the 36 or 37 sociology journals. There is a small link to it from the top 102 article post. As a side note, Distinction shows up on 15 of the top tens. The journal numbers don’t add up exactly to my final total numbers because this page was based on a slightly earlier version of the data. I’ll have to update it.

    The citation data is messy. I don’t rely on the citations counts, but instead add up all the works cited in each of the articles. I don’t know any other way to do this in automated fashion. Web of Science provides the works cited for each paper, but rarely the complete citation. The losers in this process are edited volumes that are usually not referenced in a uniform way and books with multiple editions. The winners are more recently published works that have a DOI that can uniquely identify them. But these are the same issues that flummox all citation analysis. The also misspell names, but at least their OCR program appears to occasionally misspell them consistently. All my tables and graphs are automatically generated, so I carry forward all of their quirky abbreviations and misspellings.

    At the time, I was mainly thinking about clusters of co-citations. Would they be meaningful groups of literatures? What works span different citation clusters? I’m actually more interested in betweenness centrality, which is even more heavily impact by how deeply you go into different subfields, than in total citation counts. Also, I was thinking about this as an exercise in Python programming. I wanted to release the code, and have it be used by others both for this sort of analysis, and, more generally, to learn how to do this sort of sociological analysis in Python.

    Additionally, I’m not aware of any source that looks at citations by journal or discipline. I think this is the major thing my table has to offer. To my knowledge, Google Scholar only gives all citations, and you have to do some work to get Web of Science to limit your searches. I was interested in what gets cited in sociology journals. I tried to be clear about that, writing things like, “that were cited the most in sociology journals,” but I should perhaps be clearer in the future. These are not the, “list of the 100 most cited sociology books and papers,” and I apologize for any confusion on this issue. For example, according to Web of Science, Woody Powell’s “Neither Market Nor Hierarchy” has 393 citations between 2008 and 2012, the years I looked at. According to their non-exclusive classification scheme, 171 of them were in management journals, 99 in business, 31 in economics, 27 in sociology, 20 in operations research and so on. Of the 27 that were in Sociology journals, 14 cites were in journals that are in my dataset, but my total is likely higher because they don’t count ASQ. In contrast, Mary Blair-Loy’s Competing Devotions was cited a total of 82 times, but 43 of them were in the Web of Science’s Sociology category. So over the last five years in sociology journals (as defined by Web of Science), you were more likely to encounter a reference to Competing Devotions than “Neither Market Nor Hierarchy.” The reverse is true outside of sociology.

    What my table is about, “things cited in sociology journals,” is different from “things cited by sociologists” and very different from “the number of times a work of sociology has been cited.” There’s lot of different ways to slice the pie, and I’ve tackled just one of them.

    Like

    Neal

    June 6, 2012 at 3:37 am

  7. Thanks for the explanation Neal. I think one of the main insights from the analysis is that there is a lot of sociology taking place outside of traditional sociology journals. This is especially true for subfields of sociology engaged in interdisciplinary conversations, like organizational and economic sociology, political sociology, demography, and criminology.

    Like

    brayden king

    June 6, 2012 at 1:23 pm

  8. Good point Brayden. Complicating the picture further, it is also true that many things published in sociology journals are more influential outside of sociology than in it. The most cited work published in a sociology journal in the last five years, according to Web of Science, is Chen, Curran and Bollen’s 2008 SM&R piece on structural equation models. Of its 75 citations, 72 of them were in non-sociology journals.

    Like

    Neal

    June 6, 2012 at 3:07 pm

  9. Hey, anyone at OrgTheory wanna correct the spelling of Neal’s name in that post?

    Like

    Philip N. Cohen

    June 6, 2012 at 7:42 pm

  10. Thanks!

    Like

    Philip N. Cohen

    June 6, 2012 at 9:48 pm

  11. You’re welcome

    Like

    orgtheory

    June 6, 2012 at 10:03 pm

  12. Everyone loves these kinds of lists. They make for good arguments.

    I have several reactions to the comments (including Neal Caren’s thoughtful response).

    I do think people have interpreted the list as being about the 100 most important pieces in Sociology. I do not blame this on Neal as I think he has been trying to be clear about what he thinks that means (i.e. the 100 most cited pieces that are cited in a 5 year time span by articles that appear in a purposive sample of sociology journals which he wants to say are “core” and therefore those pieces with the most cites in those core journals are the core of sociology). But, as this discussion has revealed, there has been confusion about what exactly was done here, a confusion that I hope is now resolved. What is now interesting to consider is, is Neal Caren’s interpretation of what “important pieces in sociology” one that we agree with?

    It seems to me that in constructing any kind of list, one needs to be theoretically clear about what one is interested in constructing. Every methodological decision that one makes has a theoretical implication. Neal Caren’s explanation that Woody Powell’s piece was not cited as much in his chosen group of sociology journals as Mary Blair Loy’s cuts right to the heart of the question of what is important. For me, the total number of cites a paper written by a sociologist is a better measure of its influence than the more restricted definition imposed by this analysis. But, if you think that these journals do define the core of sociology and all we should care about to define what work is important for sociology is the cites within those journals, then this list is sensible for you.

    But, I have talked to four people about this list and none of them have found it very convincing as a list of the most important pieces of sociology. Why is this the case? Sociologists do not define what is important in sociology by where the piece of sociology appears (i.e. which journals), but instead in how important a particular piece of sociology has been to sociological debate. Put another way, when I think of important pieces in substantive debates, I never think of their publication outlet as salient to the their importance in that debate. Since our subjective sense of what is important in sociology does not restrict itself to a set of publication outlets, not including all publication outlets as a source of the information for what is important in sociology (measured here by citations) certainly truncates what could go into a list of important pieces of sociology. Sociologists certainly read and cite work that is under counted by Neal Caren’s list of core journals, including work in journals in other fields. Many think this work is important or core to sociology. His method under counts these citations systematically because of the selection of his core sociology journals and it is this that makes the list he constructs seem invalid to some.

    Malcolm Gladwell has written a great piece in the New Yorker about automobile university rankings and in his usually clever way has shown that whatever you value in making the rankings turns out, not surprisingly, to have a profound effect on those rankings.

    http://www.newyorker.com/reporting/2011/02/14/110214fa_fact_gladwell

    I want to also respond to the use of network analysis here which makes really cool graphs here. The problem of the sampling is really important for network analysis. When one looks at these kinds of graph, one forgets that if a significant set of observations are missing, the entire interpretation of the graphs is a problem. In essence with incomplete data, whatever one might think of these graphs, they may or may not actually describe a set of observations and their connections accurately. So, one really needs to believe you have the population that is relevant and of course the measurement is accurate. I alert you to the work of the statistician Mark Handcock (http://www.stat.ucla.edu/~handcock/) who has actually developed statistical models of selection bias for network analysis. There are significant selection biases in these data even with the caveats provided by Neal Caren.

    Like

    fligstein

    June 6, 2012 at 11:05 pm

  13. > Hey, anyone at OrgTheory wanna correct the spelling of Neal’s name in that post?

    I’m not correcting any misspellings of Neal’s name until he learns to spell mine properly.

    Like

    Kieran

    June 7, 2012 at 12:20 am

  14. One small point of clarification. In my comment, when I compared citations to the Powell article and the Blair Loy books, I wasn’t restricting myself to the journals in my main analysis. Here, I was using all the SSCI journals, and their definition of a sociology journal. I thought that this was consistent with your original emphasis on SSCI’s list of 139 sociology journals.

    I agree that, “Sociologists do not define what is important in sociology by where the piece of sociology appears (i.e. which journals), but instead in how important a particular piece of sociology has been to sociological debate.” The list that I made doesn’t care where or when a piece was published. Of the top 10 on my list, only three appeared in sociology journals. If anything, the list shows how important books are to sociology–seven of the top ten items cited in my set of sociology journals are books.

    I also agree with you that, “the total number of cites a paper written by a sociologist is a better measure of its influence than the more restricted definition imposed by this analysis.” But I think there’s a difference between overall influence, influence within sociology and influence within a specific substantive debate. I think total citation count (what Google Scholar gives you) is a pretty good measure of overall influence, perhaps controlling somehow for variation in citing habits across disciplines. At the other extreme, looking at specific debates, examining only things cited in sociology journals also likely distorts the picture, although the extent to which this is true varies by area. Here the appropriate search strategy would probably be based on keywords rather than journals. The middle area, influence in sociology, is perhaps the least important of the three, but that’s what my table is about. If you add up all the debates that sociologists are involved in, from health inequities to social capital to revolutions to computer simulations, I’m not sure that constitues sociology, I think that’s most of the social sciences. If you restrict yourself to things published by sociologists, you miss the influence of someone like Robert Putnam or economics journals. I’d like to think that works we engage with in sociology journals are central to sociology, and that’s all I was trying to measure.

    Like

    Neal

    June 7, 2012 at 3:10 am

  15. It seems worth pointing out that to say “I don’t think about the publishing venue when thinking about the most important publications in sociology” ignores what must be a very strong selection effect. To my knowledge I have never read Anthrozoos, nor located, read, considered, or cited any article that appeared in that journal. Triply so for chapters in unknown edited volumes published by unknown publishers. I have read, and do cite, articles from other relatively minor journals such as Discourse and Society, Critical Inquiry, and the regional sociology associations’ journals.[1] But I am far less likely to spend an hour browsing the latest issue of Discourse and Society (or even Sociological Perspectives) than I am AJS or APSR. So where the item is published certainly plays an important role in the likelihood of its entering my attention-space, particularly if the item is not squarely enough within my subfield to make it into a google scholar results list.

    Like

    andrewperrin

    June 7, 2012 at 2:02 pm


Comments are closed.

%d bloggers like this: