3 ms·
It is only linear in terms of memory, not compute. Flash attention is a big advance, but not enough for 1 million tokens
by lucidrains 4y ago
It is only linear in terms of memory, not compute. Flash attention is a big advance, but not enough for 1 million tokens