4 ms·
> My experience in CS is that the replicability of experimental results is embarrassingly bad I think that's true for low-tier/low-impact CS papers, but the di
by justrealist 3y ago
> My experience in CS is that the replicability of experimental results is embarrassingly bad
I think that's true for low-tier/low-impact CS papers, but the difference is that literally nobody cares about those papers. The high-tier stuff is easily verifiable (like Tensorflow or whatever) and nobody is writing articles about 5% improvements against a benchmark in some obscure niche scheduling and planning domain.
Outside academic CS people are more scientific about experimental results because it has concrete implications on revenue or spend... but they aren't getting the results from conference papers.
- UncleMeat 3y agoI'm talking about papers in top PL and Security venues.
- wongarsu 3y agoThe ML part of CS is in a sense funding replication: there are a decent number of papers whose only premise is "we compare 5 recent papers on this benchmark (and fill a couple pages with discussion)" or "we made a new benchmark, here's how commonly cited papers compare (plus a couple pages of discussion)" Outside the high-profile cases it seems accepted norm that papers perform far worse when scored against somebody else's benchmark. The real measure of quality is how big the gap is.