4 ms·
A quick Google search reveals terms such as "sparse attention" that are used to avoid quadratic runtime. I don't know if Anthropic has revealed such details si
by FartyMcFarter 7mo ago
A quick Google search reveals terms such as "sparse attention" that are used to avoid quadratic runtime.
I don't know if Anthropic has revealed such details since AI research is getting more and more secretive, but the architectural tricks definitely exist.
- vlovich123 7mo agoThen you need to do a little bit deeper research. No one just applies sparse attention at inference time for a model not trained for it. They do this at training time because otherwise the task performance degrades too much.