There's also the problem that most complex neural networks are highly sensitive to initial weights. My friends and I have frequently tried to reproduce famous p

There's also the problem that most complex neural networks are highly sensitive to initial weights. My friends and I have frequently tried to reproduce famous papers and it's remarkable how often getting the initial settings nearly exactly correct is the key to achieving the targeted bench mark. This is a problem because cherry picking is essentially built into the frame work. If I was building ranking algorithm and just kept picking a random seed to arbitrarily sort a list of numbers until it was correct, most people would consider that obviously cheating. However if I did the same thing but stuck 3 dense matrices between the seed and the list to be ranked it would considered AI.