8 ms·
Agree. With more compute-intensive experiments in ML, I am not sure if any reviewer re-runs experiments. The reviewing is already voluntary - who's going to pay
by abhgh 3y ago
Agree. With more compute-intensive experiments in ML, I am not sure if any reviewer re-runs experiments. The reviewing is already voluntary - who's going to pay out of their pocket for the compute? It's not feasible.
- adastra22 3y agoRe-running experiments is out of scope for any reviewer. You're supposed to trust their data, but review that their conclusion is actually supported by the data, as shown.
- abhgh 3y agoYeah, I agree. But I said what I did to make the larger point that empirical studies are hard to vouch for without some level of trust on both sides. For ex if someone selects favorable datasets, metrics, "random" seeds, p-hacking etc.