4 ms·
Your comment, along with cpgxiii and n2d4’s are all really good. I have a question: suppose training and inference of an LLM were made to be deterministic at th
by dlkf 3y ago
Your comment, along with cpgxiii and n2d4’s are all really good. I have a question: suppose training and inference of an LLM were made to be deterministic at the cost of performance.
Would the cost be “everything will take twice as long” or would it be more like “inference will take a week and training will take a couple lifetimes”?
If it’s the latter, then it seems disingenuous to call this a “bug.” It’s like saying F1 cars could be horse drawn, and they only use internal combustion for “performance reasons.” If its the former, then maybe there is a more interesting discussion to be had about the potential benefits of determinism? (That said, I agree with n2d4 that it’s stupid to insult the authors. Talk is cheap and building is hard.)
- xyzzy_plugh 3y ago> That said, I agree with n2d4 that it’s stupid to insult the authors. Talk is cheap and building is hard. If your code offers an expectation of determinism then it's sloppy to not distinguish where there isn't determinism. There's nothing difficult about writing a comment to the effect of "this function is non-deterministic. For deterministic results, use X". The code is sloppy if the developers didn't consider determinism and offer nothing to consumers, or if the consumers writing software cannot know where non-determinism is introduced. If that's somehow insulting then I'd say someone has very thin skin.
- eoerl 3y agoThere are flags[1] for that indeed. It feels like half of the people commenting here don't know all that much about the topic they're commenting upon 1: https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html https://pytorch.org/docs/stable/generated/torch.use_determin...