A discussion on professional journal refereeing has appeared on numerous occasions in a variety of venues. Here's one begun on Leiter Reports (here) in 2008 that I occasionally return to. The discussion was spawned by an email to Brian Leiter from an anonymous young philosopher, and Leiter posts a snippet of the email in the post.
The young philosopher’s primary concern is how does one review a paper without being too harsh or too lenient? This is a terrific question that is largely ignored by those who are more senior in the discipline. Unfortunately, things haven't changed greatly since 2008. This just goes to show you how ill-equipped philosophy is at accommodating change. I encourage you to check out the discussion.
A commentator (J. Bogart) raises an interesting issue: should the proposed article be judged on the strength of the argument or the quality of the argument? I gather that some wouldn’t see much difference between the strength and the quality of an argument. By strength, I just mean that the reviewer agrees with what the article’s author concludes, i.e., the view the reviewer thinks is the correct view. Philosophy advances through debate. Alternative positions ought to be revealed to promote debate. So it hardly seems right for a reviewer to judge that an article be published because it’s the view with which s/he agrees.
I’ve had the misfortune of receiving reviewer comments composed of his/her opinion and not composed of some constructive criticism. For example, “I feel that the author hasn’t read x, y, and z, though these works have been cited in the references. Sure, the author makes a good argument. But I disagree with it.” Hmmm… this is a bit of juvenilia I thought I had left in my undergraduate years. But, nope, here it is in all of its glory living amongst so-called professionals. What utter crap! If I am ever tempted to write such gibberish my reports, I usually sit back and think how stupid this makes me look in the eyes of the editors of the journal. Clearly, even people who respect my views would be able to see right through such comments. They would rightfully ignore what I said and carry on with their review with other professional scholars. Equally so, they should write me off for future reviewing for them.
Why should the author care whether a reviewer disagrees with their conclusion? That's smug to believe that the reviewer's undivided attention is squarely upon them and that they are the sole determiners of the fate of the writing. I didn’t get into philosophy to agree with anyone/everyone. In fact my primary reason for getting into philosophy was to be disagreed with.
Don’t get me wrong, opinions of agreement and disagreement are important. Reviewers should be cognizant of the publication’s aims and goals. If the article doesn’t fit the aims or goals of the journal, then it shouldn’t be published in it. But a lot of what passes as being worthy of publication in the best journals fails to live up to such substantive requirements as passing the aims and goals of the very journal in which it is published. In fact, I have found that being a subscriber to the Philosophical Review has given me some insight as to how or what reviewers are looking for. The submissions must reflect the interests of the editors and the editorial assistants. Does that mean that we should begin writing things for journals that pass this "litmus test."
Reviewers — if they don’t already know what the publication’s aims and goals are — should familiarise themselves with them. Check the journal’s website, read a few articles that have appeared in the journal, and scan the table of contents for the journal over the last ten years (or so) if the reviewer’s not already familiar with the content. Reviewers should alert the author if the article’s content is not conducive with the mission of the journal. And — more importantly — the reviewer should recommend other journals more fitting for the article.
An article containing a “quality” argument is one that has fewer gaps than what you might find in a conference paper or dissertation. Similarly, the author should attempt to address any gaps s/he believes others would find in his/her argument. To not address worries anyone might have with the argument would be detrimental for the paper.
A good review evaluates the merits of the argument in the paper. Such a review, however, shouldn't evaluate the paper based upon what the reviewer wants the author to say in the paper. The review should meet the author where they're at. The review should ignore his/her own opinion on the subject when deciding whether the article is worthy of publication.
Brian Weatherson, of TAR, has published a few more comments about refereeing professional journal articles (here). His discussion is well worth more than a passing glance.
One issue that deserves serious consideration is the one that concerns how many papers are being submitted to professional journals for review. It's almost a universal assumption that there are too many papers being submitted for consideration. Then again, there are more professional philosophers available than ever before. So I don't see why there is a problem with reviewing. To the best of my ability, I think the main issue is editors' reluctance to call upon reviewers that they don't know, either personally or professionally. This, however, strikes me as something deeply sexist, misogynistic, and structurally prejudiced about the state of the philosophy profession. People are willing only to use those people that they know. Of course, they'll point out that they have to trust that referees will do a good job and those who do a good job are those that they trust. Again, this is merely a reflection of the deep seated prejudices of the discipline.
In point 6, Brian writes:
But I really don’t think the comments thread at Leiter is taking seriously how much of the problem is caused by there being too many papers being submitted… Lots of papers I see to referee are basically glorified blog points that don’t attempt to make more than a very small point. Some of them would be quite good blog posts. But most journals aim a little higher than that… I do think that many papers get sent out when the author could profitably have either rolled the paper into a larger paper, or spent time talking to colleagues/friends/blog readers about relevant literature that should be consulted.
I am often criticized by colleagues for not sending my papers out to journals for publication; instead, I choose to submit them to conferences, pass them around to folks I know and trust, or mull over them for a while. People commonly say that I’m wasting my time. If I’m understanding Brian’s point, it’s that we should develop our papers and our arguments before we send them off to a journal. A well-written, well-argued, well-developed, and well-received paper published in a journal is better than a paper that has none of these attributes.
I also have to say that much of what I have seen published recently isn’t nearly as good as it could have been if the author mulled over it for a few more days, weeks, months, years, etc. Much of the time I think that the author could have developed x line in his/her argument to make it a better paper. Shouldn’t my reaction to published papers be more like, “I don’t agree with x line of argumentation for 1, 2, 3, reasons?”
Many of my fellow ‘08-’09 job marketeers will think that much of what I’ve said is bunk (they probably won’t dismiss what Brian Weatherson’s said b/c he’s Brian Weatherson; j/k). We cannot expect to get a job if we haven’t published an article or two. I think that it’s naive to think departments will not hire an applicant just because that person hasn’t published anything. The basis of hiring depends on not only what you have done but what you are capable of doing. A short publication record should not be blamed for one’s lot in the job market.