3 ms·
As much as I liked the team, there is really no information other than the loss graph :(
by liuliu 2y ago
As much as I liked the team, there is really no information other than the loss graph :(
- az226 2y agoThat’s not quite true. They also tested benchmarks and compared with an AdamW trained model.
- bugglebeetle 2y agoNot reproducible, so this doesn’t exist.
- arilotter 2y agowe're excited to drop the paper & code and let you run the code for yourself to see otherwise !! coming in the next couple months :)
- nickpsecurity 2y agoLook at the PDF in simonw’s first link. There’s plenty of information. One part looked like the communication requirements were reduced down to under 100MB. That suggests a communication rate that could be handled by dirt-cheap instances spread across the globe. Like on vast.ai or something.
- liuliu 2y agoGaLore can do that too (if you transplant it). There are similar methods prior LLM era doing the same. They are not quite there on the loss graph side though (I am actually unsure about GaLore, but both FLoRA and ReLoRA were not quite there on loss graph side).