6 ms·
Furious AI researcher creates a list of non-reproducible machine learning papers
- lunixbochs 6y agoIt's also common for papers with code to not generalize well at all.
- htrp 6y agoLet's start with getting code, then working code, before we start complaining that your new proposed novel method doesn't generalize well
- mensetmanusman 6y agoIt’s not science if it’s not reproducible.
- nmfisher 6y agoThis isn't necessarily a recent development either. I tried to reproduce Ghahramani's factorial HMM* directly from his 1997 paper, and ran into the same problem as the other implementation I found (basically, exploding mean field values or ill-conditioned covariance matrices). By luck, I managed to find his original Matlab code and discovered he used a few small hacks (mostly re-scaling and reconditioning various values) to get it all to work. I have no problem with that, but I simply cannot understand why you would omit these from the paper (though I can think of one reason - because it ruins your elegant algebra). There is no way anyone would be able to reproduce his results if the code disappeared. https://github.com/nmfisher/fhmm_julia/blob/master/structured_vfhmm2.jl https://github.com/nmfisher/fhmm_julia/blob/master/structure...
- throwaway29303 6y agoThis is probably why the AI field will undergo a new winter.
- phreezie 6y agoThis made me think of this excellent read: https://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off https://www.newyorker.com/magazine/2010/12/13/the-truth-wear...