4 ms·
There really is a lot of cherry picking, etc. going on in this area. Papers released without code and weights or even data make reproduction and validation nea
by dunefox 4y ago
There really is a lot of cherry picking, etc. going on in this area. Papers released without code and weights or even data make reproduction and validation nearly impossible.
- forgingahead 4y agoYeah it's stunning to me that people can apparently run experiments with code, produce results with code, write a paper about that code, and then release poorly written prose in a garbled way without also releasing that code (or at the very least releasing a video demonstrating results).
- kfarr 4y ago...And get a PHD for it!
- the_only_law 4y agoI was under the impression this is just how academia worked nowadays.
- harha 4y agoThere are really just two situations if the solution generalizes well - and if it doesn’t it might be worth just mentioning that and to move on: 1. similar open data exists, great, just publish a sample implementation 2. if not the first task is to generate such an open data set Edit: formatting
- deleted 4y ago[deleted]
- time_to_smile 4y agoThere's also the problem that most complex neural networks are highly sensitive to initial weights. My friends and I have frequently tried to reproduce famous papers and it's remarkable how often getting the initial settings nearly exactly correct is the key to achieving the targeted bench mark. This is a problem because cherry picking is essentially built into the frame work. If I was building ranking algorithm and just kept picking a random seed to arbitrarily sort a list of numbers until it was correct, most people would consider that obviously cheating. However if I did the same thing but stuck 3 dense matrices between the seed and the list to be ranked it would considered AI.