so a recent posh conference just got super strict about not revealing anything about work under submission including open repository pre-print versions or even discussion on social media.
the obvious intended goal here is to re-enforce fairness in peer review, but I think this is
a) way over the top and
b) has serious consequences in terms of fairness
firstly, really significant work often is part of a body of work and experiments in guessing who did the work in at least 1 major conference that does double-blind reviewing were upwards of 50% accurate, simply because anyone who's technically knowledgeable should be aware of the work going on in the field.
secondly, researchers like to air their ideas early to get feedback before investing a lot of effort on the big project to really make an impact. so we have a whole bunch of mechanisms for this, including giving talks in seminar series, organising entire week-long retreats in cool places like Schloss Dagstuhl or Bertinoro or Tahoe, where people discuss partially formed notions in a friendly (i.e. non competitive and non-plagiaristic, collegiate) social setting.
This argues that really the extreme version of double blind is both impossible (and unfair) but also counter to the entire way science openly progresses.
we need to come up with another way to ensure fair peer review - my personal favourite is to have completely open peer review (i.e. reviewers sign reviews, authors are known) and iterative process (already used in quite a few top conferences now) where revisions allow progress, but require visibility to make the system converge.
the obvious intended goal here is to re-enforce fairness in peer review, but I think this is
a) way over the top and
b) has serious consequences in terms of fairness
firstly, really significant work often is part of a body of work and experiments in guessing who did the work in at least 1 major conference that does double-blind reviewing were upwards of 50% accurate, simply because anyone who's technically knowledgeable should be aware of the work going on in the field.
secondly, researchers like to air their ideas early to get feedback before investing a lot of effort on the big project to really make an impact. so we have a whole bunch of mechanisms for this, including giving talks in seminar series, organising entire week-long retreats in cool places like Schloss Dagstuhl or Bertinoro or Tahoe, where people discuss partially formed notions in a friendly (i.e. non competitive and non-plagiaristic, collegiate) social setting.
This argues that really the extreme version of double blind is both impossible (and unfair) but also counter to the entire way science openly progresses.
we need to come up with another way to ensure fair peer review - my personal favourite is to have completely open peer review (i.e. reviewers sign reviews, authors are known) and iterative process (already used in quite a few top conferences now) where revisions allow progress, but require visibility to make the system converge.
No comments:
Post a Comment