4 ms·

It is very impressive. However, the fact that it manages a buffer of something like 30k tokens (or whatever new limit it is) means that it's definitely autocomp

It is very impressive. However, the fact that it manages a buffer of something like 30k tokens (or whatever new limit it is) means that it's definitely autocomplete and not something with reasoning. There will be a lot of things it can't do because of that limit. That limit will also increase the amount of forgetfulness GPT4 applications have because companies using it will "auto-magically" work around the limit by sending different chunks of history to it to get whatever from it. That also means that GPT4 applications will generally be architected to have that limit - which means weirdness - non-coherent completions or code generations that "forget" something important. This is very different from real reasoning that has unlimited (yet imperfect) memory.

dougmwne 4y ago
What the heck is real reasoning with unlimited memory? Human minds are quite limited! Our short term context length is certainly not 30k words. Most psychological research says it’s about 7 items.
- pwinnski 4y ago
  Wow, this is conflating things that are not the same in an incredibly misleading way. Hey, that was 15 words, and I had them all in my head at once before I started typing! I was able to form a sentence of more than seven words all by myself! You seem to be misapplying the concept of working memory, where seven (plus or minus two) is about arbitrary and unrelated words or numbers, not "items." In fact, it's generally seven for numbers, but lower for words. And again, that's unrelated words, like "person woman man camera tv," but probably even more unrelated.
  - dougmwne 4y ago
    Did you really have those 15 words in your head all at once or did you just predict 15 next words that would form a good reply to my words?
    - pwinnski 4y ago
      Again, that's not how "working memory" works in the research. "Working memory" is considered over a time span of minutes, not fractional seconds.

dougmwne 4y ago

What the heck is real reasoning with unlimited memory? Human minds are quite limited! Our short term context length is certainly not 30k words. Most psychological research says it’s about 7 items.

pwinnski 4y ago
Wow, this is conflating things that are not the same in an incredibly misleading way. Hey, that was 15 words, and I had them all in my head at once before I started typing! I was able to form a sentence of more than seven words all by myself! You seem to be misapplying the concept of working memory, where seven (plus or minus two) is about arbitrary and unrelated words or numbers, not "items." In fact, it's generally seven for numbers, but lower for words. And again, that's unrelated words, like "person woman man camera tv," but probably even more unrelated.
- dougmwne 4y ago
  Did you really have those 15 words in your head all at once or did you just predict 15 next words that would form a good reply to my words?
  - pwinnski 4y ago
    Again, that's not how "working memory" works in the research. "Working memory" is considered over a time span of minutes, not fractional seconds.