4 ms·
For a transformer, context is already always being repeated every token. They can fetch information that became useful anytime they want. I don't see what probl
by famouswaffles 2y ago
For a transformer, context is already always being repeated every token. They can fetch information that became useful anytime they want. I don't see what problem there is to solve here.
- tsimionescu 2y agoFor a transformer, context is limited, so the same kind of problem applies after you exceed some size.