3 ms·
I am just in the process of digging into this paper and covering it in an article, so I'm quite familiar with it. But as bad as this is: What the COMPare proje
by hannob 8y ago
I am just in the process of digging into this paper and covering it in an article, so I'm quite familiar with it.
But as bad as this is: What the COMPare project is doing here is documenting the flaws of a process to counter bad scientific practice. The reality in most fields (including pretty much all of CS and ML) is that no such process exists at all, because noone even tries to fix these issues.
So you have medicine where people try to fix these issues (and are - admittedly - not very good at it) versus other fields that don't even try.
- mistrial9 8y agoits quite a stretch to say that CS does not have reproducibility ! .. probably want to define that a bit
- bjourne 8y agoThis is definitely true. A common "blueprint" for articles in applied CS is first they propose a "novel" algorithm. This algorithm may be very similar to an existing algorithm and in many cases is identical to one. Then they benchmark the algorithm and shows that it performs better on some metrics than existing solutions. These benchmarks are often quite poor, and if you vary them a little, the purported performance increases vanishes.
- Amygaz 8y agoAre you planning on touching on autoML? Do you think that could help?