3 ms·
real
by nee1r 7mo ago
real
- sheepscreek 7mo agoAre you guys affiliated with Meta’s ex-CTO in any way? I remember he famously implied that LLMs hyped. The demos are very impressive. Does this use an attention based mechanism too? Just trying to understand (as a layman) how these models handle context and if long contexts lead to weaker results. Could be catastrophic in the real world!
- sheepscreek 7mo agoI think in the long run, we may need something like a batch job that compresses context from the last N conversations (in LLMs) and applies that as an update to weights. A looser form of delayed automated reinforcement learning. Or make something like LoRA mainstream for everyone (probably scales better for general use models shared by everyone).