2 ms·
And, the attention mechanism scales quadratically with context length. This is where all of the insane memory bandwidth requirements come from.
by bob1029 2y ago
And, the attention mechanism scales quadratically with context length. This is where all of the insane memory bandwidth requirements come from.