Good article and very interesting idea! I support any idea that rewards good behavior and punishes bad in a concrete and transparent way.
One suggestion I would like to propose to further increase transparency would be to require a summary from a reviewer explaining their position and why they rejected or approved the paper.
In the "Proposed System - Reviewers' Incentives" section, you say "Potentially focus on quantity over quality to maximize earnings". I think it should be the other way around?
Very interesting ideas, unfortunately I see too many ways a payed reviewing process could degenerate :(
I have been reviewing plenty of papers in my career, always for free, always because I felt it was the right thing to do since, you know, if you want to publish a paper someone needs to read and review it (as you rightfully say in your article).
Reviewing is an act that you should do because you know it is the right thing to do, not because it is somehow useful/profitable to you. And reviewing as best as you can should be implicit, one should not need any incentive.
It is the whole scientific publishing world that should be reformed, unfortunately I do not have any expectation on that :|
I agree that it would be best if there were enough people like you who review plenty of papers thoroughly because they believe in it!
However, it is my firm belief that, unfortunately, relying on the goodheartedness of many people usually does not work sustainably. This is why in my opinion we need sustainable systems which incentivize people to do the right thing.
I am also very disappointed and disillusioned by the current system, especially also with the journals making huge profits based on free labor from academics. I agree that we need fundamental changes; with this blog post I aim to start a discussion around how such changes could look like.
I really like how recently an entirely new conference on Language Modeling was started (https://colmweb.org/). To my knowledge the first instantiation was a success. Maybe a new conference using some of these ideas can be started as a trial.
Good article and very interesting idea! I support any idea that rewards good behavior and punishes bad in a concrete and transparent way.
One suggestion I would like to propose to further increase transparency would be to require a summary from a reviewer explaining their position and why they rejected or approved the paper.
Thoughts?
I definitely agree. I think similar to the way reviews are done in the field now, it makes sense that reviewers should justify their decision.
Paying reviewers for speed: https://www.aeaweb.org/articles?id=10.1257/jep.28.3.169
Charging authors to submit, including penalties for submitting bad papers: https://www.nber.org/system/files/working_papers/w28198/w28198.pdf
Very cool, thanks for sharing these!
In the "Proposed System - Reviewers' Incentives" section, you say "Potentially focus on quantity over quality to maximize earnings". I think it should be the other way around?
Very interesting ideas, unfortunately I see too many ways a payed reviewing process could degenerate :(
I agree that it should be the other way around. This is a fear that I have with the proposed system. However, I discuss methods for fixing this issue here: https://joelniklaus.substack.com/i/151030529/new-protective-measures.
Thanks. For sure there are challenges. What do you think are the issues? Happy to discuss further.
I have been reviewing plenty of papers in my career, always for free, always because I felt it was the right thing to do since, you know, if you want to publish a paper someone needs to read and review it (as you rightfully say in your article).
Reviewing is an act that you should do because you know it is the right thing to do, not because it is somehow useful/profitable to you. And reviewing as best as you can should be implicit, one should not need any incentive.
It is the whole scientific publishing world that should be reformed, unfortunately I do not have any expectation on that :|
I agree that it would be best if there were enough people like you who review plenty of papers thoroughly because they believe in it!
However, it is my firm belief that, unfortunately, relying on the goodheartedness of many people usually does not work sustainably. This is why in my opinion we need sustainable systems which incentivize people to do the right thing.
I am also very disappointed and disillusioned by the current system, especially also with the journals making huge profits based on free labor from academics. I agree that we need fundamental changes; with this blog post I aim to start a discussion around how such changes could look like.
I really like how recently an entirely new conference on Language Modeling was started (https://colmweb.org/). To my knowledge the first instantiation was a success. Maybe a new conference using some of these ideas can be started as a trial.