I'm a physicist and a see a few problems with this. Currently, refereeing is essentially community service. Have you talked to any editors at journals such a

I'm a physicist and a see a few problems with this. Currently, refereeing is essentially community service. Have you talked to any editors at journals such as the Physical Review or New Journal of Physics? I would imagine that the largest cost of running a journal (let's say an online journal) are the costs of the editors salaries. Let's say that you scale your referee system (which is cute) and add in statistics on referees (do they often accept/reject? Areas of expertise? Time to submit a review, etc.), your editor will still have to take time to resolve the inevitable disputes between authors and referees. As the number of papers grows, you will need to either have more editors, or start paying editors to do this full time. Another problem that you will run into is the question of curated content. There is a place for having refereed content that is technically correct, but not necessarily impactful (for example, Nature's scientific reports), but such a journal will have a low impact and will have a hard time attracting impactful papers (which are still valuable for career advancement). Otherwise, your editors can start to curate the content based on impact, but this requires even more time and expertise from your editors. I think you'll eventually run into some of the same problems as traditional journals... Have you thought about working with existing journals to improve the refereeing process? For some of the other issues that you bring up, such as attaching code--I think that this will have to be something required by funding agencies. I have found the "new" supplementary material sections of journals to be a source of improvement...