3 ms·
https://ml-gsai.github.io/LLaDA-demo/ https://ml-gsai.github.io/LLaDA-demo/ As far as I understand (based on reading for less than 5 minutes) it is still a tra
by imtringued 2y ago
https://ml-gsai.github.io/LLaDA-demo/ https://ml-gsai.github.io/LLaDA-demo/
As far as I understand (based on reading for less than 5 minutes) it is still a transformer model, but they simply start predicting tokens in random positions, with the possibility of updating existing tokens, rather than producing tokens from left to right.
That isn't too different from say good old stable diffusion.
- clbrmbr 2y agoThat sounds like it would be far less computationally efficient. How can there be an efficiency gain over autoregression?
- emporas 2y agoGPUs compute in two dimensional vectors, and auto-regressive models compute in sequence. Diffusion seems to me to be the more efficient and natural fit for computation on GPUs. When o3 solved the ARC[1] challenge, Sam Altman said it was recognizing the image pixel by pixel linearly, and i found it pretty hilarious that they start by using a chip which computes in two dimensions, they use it to compute a line, and that line is a two dimensional image of the ARC challenge. [1] https://arcprize.org/ https://arcprize.org/