3 ms·
Why isnt LLM training itself open sourced? With all the compute in the world, something like Folding@home here would be killer
by andoando 6mo ago
Why isnt LLM training itself open sourced? With all the compute in the world, something like Folding@home here would be killer
- DesaiAshu 6mo agodata bandwidth limits distributed training under current architectures. really interesting implications if we can make progress on that
- dogcomplex 6mo agoLimits but doesn't prohibit. See https://www.primeintellect.ai/blog/intellect-3 https://www.primeintellect.ai/blog/intellect-3 - still useful and can scale enormously. Takes a particular shape and relies heavily on RL, but still big.
- andoando 6mo agoWhat bandwith limits? Im assuming the forward and backward passes have to be done sequentially?
- DesaiAshu 6mo agoYes also passing data within each layer
- throwaway27448 6mo agoIt's either illegal or extremely expensive to source quality training material.
- m4rtink 6mo agoYeah, turns out if you want to train a model without scrapping and overloading the whole of Internet while ignoring all the licenses and basic decency is actually hard & expensive!
- mike_hearn 6mo agoIt is in some cases. NVIDIA's models are open source, in the truest sense that you can download the training set and training scripts and make your own.
- doctorwho42 6mo agoWell it is, it's in the name "OpenAI". /S