4 ms·
> I do ML every day. > I go to the github. Maybe model download link is there. I see zero code Paper was released today. Chill. They said they will release t
by godelski 5y ago
> I do ML every day.
> I go to the github. Maybe model download link is there. I see zero code
Paper was released today. Chill. They said they will release the code in September (I'm guessing late September). The paper is also a pre-print. They're probably aiming for CVPR and don't want to get scooped.
> Model first. Code second. Paper third.
That's how you produce ML code and documentation but that is not how you release it. I guarantee you that they are still tuning and making the model better. They're were still updating ADA till pretty recently (last commit on the pytorch version is 4 months ago, to code).
I originally wasn't in CS, and when I first came over I wasn't in ML. We never had code. The fact that ML publishes models AND checkpoints is a godsend. I love it. Makes work so much easier and helps the community advance faster. I love this, but just chill. The paper isn't peer-reviewed. It is a pre-print. They're showing people what they've done in the last 6 months. It's part publicity stunt, part flex, part staking claim, but it is also part sharing with the community. Even without the code we learn a lot because they attached a paper to it. So chill.
- sillysaurusx 5y agoThe fact that you had to say "chill" three times indicates that you're trying to convince yourself, not me. None of what you said is responsive to what I wrote. I think it's an opinion piece, but I'm not sure. The issue here is the scientific method. I've listed the things that are required, as I see it. And I've also listed the reasons why I haven't been able to verify it exists here, despite trying for two years. I'm glad that you like ML hacking, and I like it too. But models aren't a godsend; they're "the most basic, bare-minimum requirements of reproducibility." Your reaction shouldn't be "I'm incredibly grateful you'd be willing to do this." It should be "You're required to do this, because if I can't verify your claims, your claims might be mistaken." To leave it off on a softer note, normally I'd bond with you, ML hacker to ML hacker. Because I love ML, and I love hearing what you've been up to in ML. It's the best job in the world, as far as I'm concerned. (Could any other career give you the opportunity to be a developer advocate for high-performance computing in such an interesting way? https://github.com/google/jax/issues/2108#issuecomment-866238579 https://github.com/google/jax/issues/2108#issuecomment-86623... Definitely looking for more examples of "Github Larping," if you know of any.) If you agree that the scientific method is the reason ML moves forward, all I'm doing here is protecting it.
- taneq 5y agoThe repetition here is a common rhetorical device, not necessarily an indication of self-doubt. That said, I agree with your overall position on ML publications. So much of what we see is a tech demo protected by some kind of moat, either a private commercial dataset or insatiable processing requirements or missing code or a combination of the above. These aren’t science, they’re advertisements.
- godelski 5y agoI say chill because you're freaking out. The scientific method is being followed here. Code is not needed for the scientific model to be followed. Even data. Literally every other field is able to advance without public code or data (in fact most areas of CS). There's absolutely no reason to believe that they won't release their code. They have a history of doing so. Models and checkpoints are not the bare-minimum for reproducibility. They describe their model enough in the paper. There's enough written in the paper (which is 30 pages) to reproduce the model. Will it be easy? No. But it can be done. And to be clear, I'm saying that the status quo of code being released is a godsend. This is not the norm in literally every other field/subfield. Code helps with reproducibility (and so should be encouraged) but is not required. If you require someone else's code to reproduce results then you're not convincing me you're a good ML researcher nor programmer.
- z3ncyberpunk 5y agoNo one is freaking out... you are projecting.
- deleted 5y ago[deleted]
- sillysaurusx 5y agoI'd be no scientist if I didn't update my priors. I retract my claims. You're right. Thanks for calling me out. I will say that it's... a gargantuan effort to do the things that you're proposing. But as someone who did them you're right, you can. (BigGAN-Deep took a year to track down the bug https://github.com/google/compare_gan/issues/54 https://github.com/google/compare_gan/issues/54) BigGAN-Deep is a decent example of the thing I was really worried about: replication. I thought it'd be really easy to "just implement the paper." But no one had. Mooch did, but not at the same scale as the DeepMind release. Maybe you're right about me, too. You're convincing me that I'm not a very good ML programmer. It's probably best to bow out on whatever high notes I've achieved. Karras' work is fantastic. I don't know why this preview of things to come was where I chose to do this. Thank you, nVidia group, for working so hard.
- black_puppydog 5y ago> They're probably aiming for CVPR The template screams NeurIPS though. Page limit for that would be 9 pages, this is 9.5, they might have started adding things after the first deadline, anticipating an extra page for camera ready? I mean, that's just a bit of paper astrology of course. But if I'm right, then the author notification is September 28 and camera ready will be due in October, assuming it is accepted. So in that case releasing code (end of) September makes sense. Edit: regardless of the (good) work NVidia have been doing over the last years, there is an issue here about big teams breaking the blind review process by putting themselves on the front of not just HN, but by now probably also the relevant twitter, fb, reddit pages. They know full-well that a release by NVidia will gain attention, and by the time review really gets started it's very likely any reviewer in their field will know exactly who they're reviewing.
- godelski 5y ago> The template screams NeurIPS though. Page limit for that would be 9 pages, this is 9.5, That's a fair point and I'm not sure why I didn't consider that they would release a pre-print after they had submitted it. (This is a total fumble on my part) > there is an issue here about big teams breaking the blind review process by putting themselves on the front I don't see that as actually breaking the blind review part. There are many more abuses that de-anonymize themselves. Most transformer research is done by big labs because they need the processing power and are the only ones who can afford such equipment (though there was a paper that did transformers on CPUs). Just training ImageNet is out of bounds for a lot of people (I have a few A6000s and it still takes me days). A trivial example is that Google will use JFT and will include it everywhere. If you're qualified to review you're probably going to be able to de-anonymize the lab. I do think we need to do more to make a more level playing field but that's an extremely difficult thing to do. More resources just enables you do do more. But maybe we shouldn't metric hack as much, which would slow things down a little.