I recently submitted a comment to Phys. Rev. Lett. (which you can find on arXiV[1]) in response to a paper published a few weeks ago. This is only the second comment I’ve submitted in 17 years as a scientist; in both cases, I wanted to correct what I saw as a serious flaw in the way the field in question was being presented. But is this the right way to engage with what we see as problems with a paper? The idea of post-publication comments or peer-review, which could be an alternative, is gaining some currency.
The bedrock of academia has been, until recently, the idea of critical but fair peer review. Of course, this requires some dedication on the part of reviewers, and that work is slowly being given recognition (see, for instance, the APS Outstanding Referee programme). There is also a question about the effort put into refereeing by academics at different stages of their careers – some correspondence in Nature recently has suggested that senior academics ought to referee at least as many papers as they submit.
But what happens if refereeing allows through a paper which you feel is not right, or makes claims which are not justified ? The traditional model is to write a comment, which is then sent out for review, while the authors are allowed to respond. This can be a slow process (a recent article by Philip Moriarty on an on-going dispute told of a comment that took three years to be published) and is really only a one-shot approach; there is no room for nuanced debate. (In the spirit of disclosure, the comment I submitted to PRL concerned a paper on linear scaling ab initio molecular dynamics[2] (here); I felt that the paper rather over-stated its achievements, and underplayed the achievements of others in the field. It is with the editors at present – a process that can easily take two weeks. But how many people who read the original will read the comment ?)
An alternative model for refereeing is that of PLOS One and other journals: so long as the paper is accurate, publish it, and if it is high quality then it will be widely cited. Going one step further than this, some journals are publishing referees reports along with the paper, and others are encouraging post-publishing comments – and in some cases referee-style comments and criticisms after publication. In the case of the paper I am commenting on, I think that this would be very useful. But there are (at least) two clear objections: what about impact; and what about moderation?
Over the last twenty years, impact has become more important, and this is largely measured by the Impact Factor of the journal, and the citations gained by a paper. There are have been many discussions of this problem already (and I am sure that there will be many more) but it does give the referees and editors of certain journals tremendous power over what fields are deemed important.
An alternative approach which is emerging is to account for other ways of following attention: ImpactStory will accept more than just papers, and will track on-line bookmarking, tweets and downloads to assess the broader impact of some work; AltMetric does a similar job; and many publisher websites allow you to follow the number of downloads of an article. It is clear that this gives an alternative approach to analysing impact, though it may take time to establish itself.
Moderation is harder. This is the advantage of the present system: the editor effectively moderates the referees’ comments, and the authors’ responses. Without some form of moderation, on-line commenting can swiftly degenerate, as much of the internet proves. It’s not clear to me how this will develop: possibly editorial teams could take on this role, or it could become as much a part of academic life as refereeing is now. It’s clear, however, that things are changing, and academics need to be part of the debate.
[2] Phys. Rev. Lett. 112, 046401 (2014) DOI:10.1103/PhysRevLett.112.046401