2 ms·
I'm still reading the paper, but my main question is how slow is the model compared to LLM of the same size. It seems like to get the best accuracy they need to
by kadushka 2y ago
I'm still reading the paper, but my main question is how slow is the model compared to LLM of the same size. It seems like to get the best accuracy they need to set number of time steps to the number of tokens to be generated. Does it make it comparable in speed to an LLM?
- kadushka 2y agoUpdate: finished the paper, and as I suspected, there's a serious downside in speed and memory consumption. LLaDA model has to process the entire output sequence on every time step - without anything like KV cache. Also, full quadratic attention happens on the entire output sequence on every time step, which makes it unfeasible for sequence length longer than a few thousand tokens.