3 ms·
I used to think it was the quadratic complexity of attention but I guess that's not a concern anymore as they've made more hardware aware kernels of attention?
by rishabhaiover 7mo ago
I used to think it was the quadratic complexity of attention but I guess that's not a concern anymore as they've made more hardware aware kernels of attention?
The other I remember is continual learning but that may be solved in near-term future.
I am not completely confident about it.
- ACCount37 7mo agoHumans do have an upper limit on how much working memory they have. Which I see as the closest thing to the "O(N^2) attention curse" of LLMs. That doesn't stop an LLM from manipulating its context window to take full advantage of however much context capacity it has. Today's tools like file search and context compression are crude versions of that.
- rishabhaiover 7mo agoHuman brain's prediction loop is bayesian in nature.
- rishabhaiover 7mo agoDamn, the research moves fast. I was wrong again: https://arxiv.org/abs/2507.11768 https://arxiv.org/abs/2507.11768