4 ms·
Who says GPT has the ability to hold 25,000 token in its "head"? You can send 25000 random words in the prompt and asks GPT how many pairs of words share at le
by d--b 4y ago
Who says GPT has the ability to hold 25,000 token in its "head"?
You can send 25000 random words in the prompt and asks GPT how many pairs of words share at least one letter. I doubt that the answer will be correct...
- moconnor 4y agoWhy? I'm pretty sure it could do this kind of task - attention is computed between all pairs of tokens. Yes, it's a lot of compute.
- criddell 4y agoSurely GPT could write a program to count pairs of words that share at least one letter, right? Maybe GPT-5 will be able write and run programs on the fly to answer questions like this.