3 ms·
They can learn from mistakes, within the context window. I often will correct GPT4 on a mistake it's made and then ask it to write me a new prompt from scratch
by dinobones 4y ago
They can learn from mistakes, within the context window. I often will correct GPT4 on a mistake it's made and then ask it to write me a new prompt from scratch that covers the edge case.
GPT4 has a 32k token context window. But I see this number the like the 32mb of RAM of computers of old.
With Flash Attention, the attention calculation is now basically O(n). So we could maybe someday see context windows in the billions of tokens, where "learning" is just looking back at a mistake in its past and avoiding it.
- tomohelix 4y agoGood point. I guess it is a matter of scale. Maybe when we get to something like billions of token windows or even more, it becomes "memories" for the AIs. And they can probably make better "judgements" using these memories to train next generation of AIs to internalize these improved logics without the need to have the context fed to them. Kinda similar to how we human do it too. It is certainly an exciting time that we are seeing so many parallels between silicon and meatbags. I didn't think I would live to see this.
- orbital-decay 4y agoThe main difference between ML in its current state and biological systems is the huge compute/energy asymmetry between training and inference. Biological systems have it too, but it's much less prominent, as they have neuroplasticity which allows them to learn on the fly. I guess the context window can be considered a (poor) substitute for the missing neuroplasticity, but it's really limited in what it can do.