The problems of peer review

Another good posting on the ups and downs of peer review from the group blog, New APPS: Art, Politics, Philosophy, Science.  It’s a good prompt for us to remind readers about the process of peer review at Society and Space and, while recognizing the extraordinary demands made on scholars today, to consider the importance of peer review for maintaining the excellent quality of our authors’ papers.

At Society and Space, every initial submission is typically read by all four of the editors as part of a prescreening process.  We consider the paper’s fit with the broad aims of the journal, quality of the paper, its theoretical sophistication (i.e., the suitability of the approach for our readership), its empirical rigor, the appropriateness of length and style, and whether a redirection to another journal is a better route than peer review with us, given the answers to these considerations.  We then contact the author to redirect them to another journal, or we offer advice on how to get the paper into a state suitable for review with us, or we enter the paper into peer review.

We’re constantly working on ways to improve the review process, but at the heart of the process is every reviewer’s commitment to provide timely and detailed feedback for authors.  Stuart Elden wrote an editorial in 2008 about the “exchange economy” of peer review, explaining the need to provide at least three times the reviews as one’s own submission record.  At the time he wrote the editorial, the journal received about 150 submissions per year.  Now, we receive 250+ submissions every year.  If every paper put through to peer review generates the need for at least three referee reports, that means we have to ask at a minimum 375-450 referees per year or so.

That is a best-case scenario, and often it’s a goal not realized when people commit to doing a review and then don’t follow through.  We often have to make decisions to accept or reject papers with only two reviews, but we always seek to generate a third or fourth review when two referees differ. We might wait two or three months for a review that never materializes, only to have to start over with new invitations.  It’s usual to need to ask five or six people to review to get two or three final reports.  On the other hand, it’s not at all uncommon for us to have to ask ten or more people to review a paper, just to get two productive reviews. A few papers in the past year each have needed 15-20 invitations to referees, only to generate a final two reviews.

Another obvious issue is the quality of reviews. We most often get excellent and informative reviews from our referees, and we thank them for the wonderful labor they do for the journal and for authors.  Occasionally, however, we struggle with how to frame a one-paragraph review from a referee to an author, as the brevity contains precious little information on how to improve a paper’s theoretical argument or strengthen its writing.  The commitment to review a paper can be onerous, but the result is always a better paper or in the least vital feedback so that the author can try to rework a rejected paper for another publication outlet.  The journal always benefits from detailed and rigorous reviews, as does the community of scholars who read and contribute to Society and Space, and who engage with critical social/spatial theory more generally.

On our publisher’s guidelines for authors page, we state that any submission comes with an expectation that authors will accept requests to review in the future from us. This is the suggestion that Catarina Dutilh Novaes makes at the conclusion of the APPS posting, which generated many hearty comments. We welcome yours here.


13 thoughts on “The problems of peer review

  1. stuartelden says:

    Reblogged this on Progressive Geographies and commented:

    Society and Space editor Mary Thomas picks up on a discussion at New APPS concerning peer review.

  2. Reblogged this on AESOP Young Academics and commented:
    This piece on the Society and Space website about “the problems of peer review” deserves prompt reblog, as it engages with issues that we at the YA blog are extremely interested about.

  3. sarakoopman says:

    I review for some 15 journals, and only now, with a new IR journal, have I for the first time received a fabulous email back from the editors telling me their final decision on the article and including the reviews from the OTHER reviewers. It was fabulous to get to read their takes and suggestions on the piece, and it made the whole process more worthwhile for me. I wish all journals would do this but maybe Society and Space could start?

    • stuartelden says:

      Thanks Sara. This is something that we’ve discussed before as, like you, I’ve experienced it in a few instances – but only from a very small proportion of the journals I’ve reviewed for. It would be quite a bit of additional work for the editorial team, though that might be automated. Referees may object to a decision reached on the basis of their report, which could potentially lead to correspondence. Editors are working many additional hours to edit journals, rarely adequately compensated by reduction in other work, and I fear adding to that workload. Would it increase people’s willingness to review? Really that is the key concern: getting good, timely reports to enable prompt, well-justified decisions. I’m not sure, and like you think the idea is good in principle. But I was persuaded against this by people more knowledgeable than me in the past.

      • sarakoopman says:

        hmmm. Well it seems like it could easily all be automated, so it wouldn’t lead to any additional work for editors. What I got was clearly an automated form letter. I honestly was not planning to review for that journal again but having gotten that nice letter from them will now do so, so at least for me it worked to increase my willingness to review. It seems like the chances that it would do so far outweigh the chance that it would piss someone off because they had suggested one course and the editors took another. I think we all understand that’s the role and prerogative of the editor.

      • stuartelden says:

        Thanks Sara. Yes, it should be more straight-forward with the online manuscript system than it was before that. We will discuss again.

      • Hi both. My two cents. Imagine an editorial decision (revise and resubmit) based on three really diverging reviews (accept with minor revision; revise and resubmit; reject). Let’s imagine the revised paper is then accepted: once it’s out, the reviewer who suggested rejection may anyway get to know. Having been informed of the others’ reviews, (s)he already expects this. Moreover the other’s reviews already explain why, so there is less reason for (s)he to ask justification to the editors.
        To sum up, peer review is a process where editors have a lot of power exactly because of blindness (sometimes I wonder if they have too much): I think any step towards transparence and accountability would reduce friction and improve trust, at the end of the day.

  4. natalieoswin says:

    Thanks Sara and Stuart. I too find that it makes the review experience more rewarding when informed of the outcome and provided with the other reviewers’ comments. Our editorial team last discussed this issue before we had an online submissions system. If this can be automated, it’s something we should revisit.

  5. sarakoopman says:

    Yay! For the purposes of thinking about how it might be automated let me just add that what I got from them was a short form letter, and attached was a copy of the letter they sent the author, and a copy of each reviewers submission.

    • marythomas says:

      Thank you, Sara! I agree completely with the merits of this, as it allows reviewing to be more intellectually stimulating for the referee.

  6. […] Society and Space – The problems of peer review […]

  7. […] few days ago, I had reblogged a post at the Society and Space website about “the problems of peer review”. It is a very interesting piece, whose importance is primarily that of “disclosing” to the […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: