I think this separation falls down badly in some cases: after someone claims a method worked but it's a dubious model, you can't possibly reproduce their result

I think this separation falls down badly in some cases: after someone claims a method worked but it's a dubious model, you can't possibly reproduce their results then there's no way to know that their model actually even worked once when they did it or if they just made an entirely stupid mistake (like using the complete data set for training). Concretely there's applied ML papers on confidential military data, where I'm fairly sure the results are just bogus or cherry-picked or done with incorrect procedure, but with the data being unobtainable (even to those with clearance the data isn't materialized in some DoD database for reproducing) it makes the published results entirely unreproducible and unchallengeable.