3 ms·

As someone who's often implementing models from ML papers, the issue is that english-language descriptions of methods is often found to be sorely lacking. Even

As someone who's often implementing models from ML papers, the issue is that english-language descriptions of methods is often found to be sorely lacking. Even good authors simply forget to mention small critical details of model design or training that wouldn't remain ambiguous if they simply released their code. Part of the effort of reproducing work is certainly figuring out which aspects of the model design are critical or incidental - but this is all greatly aided by not having to guess at what was actually done from a heap of english and LaTeX generally written feverishly over a few days before a deadline!

iguy 8y ago
> Part of the effort of reproducing work is certainly figuring out which aspects of the model design are critical or incidental Isn't this the work of writing up? A paper is a claim that you have discovered something, and implicitly that other details are standard / unimportant. If it turns out that some hidden assumption was in fact doing all the work, then the claim you made was false. I'm all in favor of sharing working code. But working code which magically does something... amounts to an anomaly awaiting an explanation. Or an advertisement.
- sdenton4 8y ago
  Alas, training times are quite long... At any given time, there's probably a couple best-in-class architectures out there for the problem you're interested in, and one or two dozen interesting bells and whistles one can add as decoration, each with a paper that makes pretty reasonable arguments and has some stats demonstrating modest gains. The right thing to do in this circumstance is an ablation study - throw together your best-possible model and then test different subsets of features sitting between your model and the 'basic' prior work. For large datasets, though, each of these models might take a very long time to train (especially if you don't work at a place with a stupid number of GPUs available). So, lacking resources, you get your new best-ever accuracy number with your 'everything' model, and do an extensive write-up about how awesome the new bell and/or whistle that you added to the pile is... (The problem is compounded by a need to publish quick, lest someone else describe your bell/whistle first.) Another big problem is that adding a bell/whistle to the base model often means adding more parameters to the model. There's decent evidence coming out of the AutoML world that number of parameters matters a hell of a lot more than how you arrange them. (It's real real easy to convince yourself that your clever new idea is more important than the shitpile of new parameters you've added to the model, after all.) So a really solid ablative study probably needs to scale the number of parameters in a reasonable way as you add/remove features... And there may not be obvious ways to do that smoothly. And this is closely related to the replication study in psych: it's real real easy to do kinda sloppy work with big words attached, and convince yourself and all your peers that you're a genius. I think a big database for reporting and searching for results with various architecture+dataset combinations would be much more useful than pushing more papers to the arxiv in many cases. (though, really, whynotboth.gif) Let me do some searches to see if a particular bell/whistle actually adds value across the god-knows-how-many-times someone's used it to train up imagenet from scratch...
  - iguy 8y ago
    Then it sounds like people are doing something quite far from science. That's not a criticism, there is lots of knowledge which isn't scientific... for example if you want to know how to run a restaurant, you need to talk to lots of people who do, and try to learn what they know about the art. Maybe you even go to trade shows and listen to talks. But if you follow what they said and you fail, it's not really a replication crisis.
    - sdenton4 8y ago
      It's a mix; some groups are more disciplined than others (and have better resources for ablation studies). To be sure, some sciences are more rigorous than others. Various natural sciences are reliant on transient observations, which might or might not be 'reproducible' in any sense... And still progress is made, though the results might contain more prejudices than one would encounter in other areas. It's also worth noting that some areas within machine learning are extremely difficult to test rigorously (e.g., generative processes), but are still totally worth pursuing. So be careful with cries of 'it's not even science, man!'
      - iguy 8y ago
        Right I genuinely don't mean that as an insult, perhaps my cooking example was flip. There is lots of genuine progress made by other means -- anyone think that Toyota's process inventions in were done by double-blind trials on a large sample of factories? And of course some guys a few doors down are proving theorems. And lots of other things. It is slightly strange though that we try to vet all of these with the same peer review mechanism. This is one major source of the differing opinions in this thread about how this ought to work, I think.

iguy 8y ago

> Part of the effort of reproducing work is certainly figuring out which aspects of the model design are critical or incidental Isn't this the work of writing up? A paper is a claim that you have discovered something, and implicitly that other details are standard / unimportant. If it turns out that some hidden assumption was in fact doing all the work, then the claim you made was false. I'm all in favor of sharing working code. But working code which magically does something... amounts to an anomaly awaiting an explanation. Or an advertisement.

sdenton4 8y ago
Alas, training times are quite long... At any given time, there's probably a couple best-in-class architectures out there for the problem you're interested in, and one or two dozen interesting bells and whistles one can add as decoration, each with a paper that makes pretty reasonable arguments and has some stats demonstrating modest gains. The right thing to do in this circumstance is an ablation study - throw together your best-possible model and then test different subsets of features sitting between your model and the 'basic' prior work. For large datasets, though, each of these models might take a very long time to train (especially if you don't work at a place with a stupid number of GPUs available). So, lacking resources, you get your new best-ever accuracy number with your 'everything' model, and do an extensive write-up about how awesome the new bell and/or whistle that you added to the pile is... (The problem is compounded by a need to publish quick, lest someone else describe your bell/whistle first.) Another big problem is that adding a bell/whistle to the base model often means adding more parameters to the model. There's decent evidence coming out of the AutoML world that number of parameters matters a hell of a lot more than how you arrange them. (It's real real easy to convince yourself that your clever new idea is more important than the shitpile of new parameters you've added to the model, after all.) So a really solid ablative study probably needs to scale the number of parameters in a reasonable way as you add/remove features... And there may not be obvious ways to do that smoothly. And this is closely related to the replication study in psych: it's real real easy to do kinda sloppy work with big words attached, and convince yourself and all your peers that you're a genius. I think a big database for reporting and searching for results with various architecture+dataset combinations would be much more useful than pushing more papers to the arxiv in many cases. (though, really, whynotboth.gif) Let me do some searches to see if a particular bell/whistle actually adds value across the god-knows-how-many-times someone's used it to train up imagenet from scratch...
- iguy 8y ago
  Then it sounds like people are doing something quite far from science. That's not a criticism, there is lots of knowledge which isn't scientific... for example if you want to know how to run a restaurant, you need to talk to lots of people who do, and try to learn what they know about the art. Maybe you even go to trade shows and listen to talks. But if you follow what they said and you fail, it's not really a replication crisis.
  - sdenton4 8y ago
    It's a mix; some groups are more disciplined than others (and have better resources for ablation studies). To be sure, some sciences are more rigorous than others. Various natural sciences are reliant on transient observations, which might or might not be 'reproducible' in any sense... And still progress is made, though the results might contain more prejudices than one would encounter in other areas. It's also worth noting that some areas within machine learning are extremely difficult to test rigorously (e.g., generative processes), but are still totally worth pursuing. So be careful with cries of 'it's not even science, man!'
    - iguy 8y ago
      Right I genuinely don't mean that as an insult, perhaps my cooking example was flip. There is lots of genuine progress made by other means -- anyone think that Toyota's process inventions in were done by double-blind trials on a large sample of factories? And of course some guys a few doors down are proving theorems. And lots of other things. It is slightly strange though that we try to vet all of these with the same peer review mechanism. This is one major source of the differing opinions in this thread about how this ought to work, I think.