4 ms·
While the race to incorporate longer and longer context (2K PaLM -> 32K now) is interesting, I don't think that'll scale. It'll just add too much noise to the h
by 1024core 4y ago
While the race to incorporate longer and longer context (2K PaLM -> 32K now) is interesting, I don't think that'll scale. It'll just add too much noise to the history: how do you establish a causal relationship between what you're holding in your hand, versus the million other things (context) that you've seen in the past. You'll end up with spurious correlations.
What I think (and this is just me talking out of my ass) will be required is some form of associative long-term memory. Basically, give the model a way to store some embeddings in some form of memory, and then retrieve them based on context: so it doesn't matter if you encountered that item 2 tokens ago, or 2B.
At least this is what my current intuition tells me.
- lucidrains 4y agothat line of research is still going. https://github.com/lucidrains/block-recurrent-transformer-pytorch https://github.com/lucidrains/block-recurrent-transformer-py... i think it is worth continuing research on both fronts.
- 1024core 4y agoOf course, I'm not saying "don't do research". I'm just saying that I don't think this context-length war will lead us to long-term sustainable gains.
- deleted 4y ago[deleted]
- nathias 4y agothis and evaluation based on context that lets it mutate past content and we are set
- skybrian 4y agoOn the other hand, it seems like training on large amounts of text for next-token prediction would tend to reduce reliance on spurious correlations? I don’t think this intuitive sort of speculation can predict what it will do.