4 ms·
ML researchers are playing scientists: tweak a few parameters in an LLM, re-train it on a largish dataset (need access to $$$ GPUs), find metrics on which the t
by akomtu 3y ago
ML researchers are playing scientists: tweak a few parameters in an LLM, re-train it on a largish dataset (need access to $$$ GPUs), find metrics on which the tweaked LLM makes a barely noticeable improvement and make the other metrics where it actually gets worse look insignificant, write a paper, upload to arxiv, and update your resume.
- brucethemoose2 3y ago> write a paper Also publish yet another "SOTA framework" thats barebones and won't be maintained for very long! The researchers who published the negative CFG LLM paper made an earnest effort to pull it into the popular frameworks. That really stands out in my memory, that is incredibly rare.
- jsemrau 3y agoWorse are comparison papers where the researchers are evaluating models against a benchmark. In another world, this would have been a blog post.
- zaptheimpaler 3y agoEvery real ML position industry or academia explicitly asks for publications. The field gets what it asks for, and they get it good and hard.
- Tostino 3y agoI'd rather the knowledge be out there, it helps you not go down dead end paths that you would otherwise have explored.
- marcinzm 3y agoSo true, if they were real scientists then they'd never publish negative results and only pass them along in their network to most efficiently gate keep the field!
- timkam 3y agoThis paper presents formal results, apparently. Also, in the case of formal results, peer review by experts in the exact sub-field makes sense/increases trust, sure, but why not share on archive instead of waiting for a year (or however long the process takes in the particular instance)?
- sebzim4500 3y agoI genuinely can't tell if this is supposed to be a criticism. They tried something, it improved some things and made other things worse.