heyitsguay is correct, but in addition: - The researchers didn't even explore quantization. In theory 4 bit quant would allow for training on even more modest

heyitsguay is correct, but in addition: - The researchers didn't even explore quantization. In theory 4 bit quant would allow for training on even more modest hardware. - Memory use aside, forward pass only is potentially a big training speed increase. - This method is very amenable to decentralized training. It scares me because it feels like a gateway to networked, self training LLMs on commodity hardware. I thought this was a long way away... now it doesn't feel so far away.