4 ms·

Can you memorize “War and Peace” in one sitting and randomly and precisely recall things from it on demand? No? Didn’t think you could. Could you perhaps transl

by ein0p 2y ago

Can you memorize “War and Peace” in one sitting and randomly and precisely recall things from it on demand? No? Didn’t think you could. Could you perhaps translate Farsi into Hawaiian and then speak it? Can you give instant, and nearly always correct to almost any textbook question irrespective of the domain? All of that is recall, and all of that GPT4 could do for well over a year.

refulgentis 2y ago
> Can you memorize “War and Peace in one sitting - neither can an LLM, unless you weaken it to "well I meant input that fits in context", not memorize > precisely recall things from it on demand - "precisely" is load-bearing here, and the question at hand. (no, LLMs don't have perfect or precise recall, even from in-context material, and it's not particularly close) > translate Farsi into Hawaiian and then speak it - 0 models, is this supposed to be a gpt-4o reference? What does this have to do with "perfect recall?" Let's pretend it does. Are you claiming that gpt-4o has a 100% success rate translating Farsi to Hawaiian? > Can you give instant, and nearly always correct to almost any textbook question - sobs in just spent 4 days benchmarking. For the record, best non-RAG I have is gpt-4o at 86% on MedQA and LegalBench. With RAG, 93%. RAG gpt-4o just barely scrapes by my cofounders USMLE scores. It's excellent. But it's not superhuman. > All of that is recall - no, translation is not recall
- imtringued 2y ago
  You're quoting yourself. It's hard to understand which quotes are yours or not.
  - ein0p 2y ago
    Ironically, a failed recall issue. They forgot they needed to add newlines in between. :-)
- ein0p 2y ago
  But you see, while some of those things aren’t _just_ recall, they are recall, too. Just different types of it, rather than point recall. And a modern SOTA LLM can accurately recall _vastly more_ data than any human or even any group of humans. Transformers are quite literally very fancy associative memories.

by ein0p 2y ago

refulgentis 2y ago

> Can you memorize “War and Peace in one sitting - neither can an LLM, unless you weaken it to "well I meant input that fits in context", not memorize > precisely recall things from it on demand - "precisely" is load-bearing here, and the question at hand. (no, LLMs don't have perfect or precise recall, even from in-context material, and it's not particularly close) > translate Farsi into Hawaiian and then speak it - 0 models, is this supposed to be a gpt-4o reference? What does this have to do with "perfect recall?" Let's pretend it does. Are you claiming that gpt-4o has a 100% success rate translating Farsi to Hawaiian? > Can you give instant, and nearly always correct to almost any textbook question - sobs in just spent 4 days benchmarking. For the record, best non-RAG I have is gpt-4o at 86% on MedQA and LegalBench. With RAG, 93%. RAG gpt-4o just barely scrapes by my cofounders USMLE scores. It's excellent. But it's not superhuman. > All of that is recall - no, translation is not recall

imtringued 2y ago
You're quoting yourself. It's hard to understand which quotes are yours or not.
- ein0p 2y ago
  Ironically, a failed recall issue. They forgot they needed to add newlines in between. :-)
ein0p 2y ago
But you see, while some of those things aren’t _just_ recall, they are recall, too. Just different types of it, rather than point recall. And a modern SOTA LLM can accurately recall _vastly more_ data than any human or even any group of humans. Transformers are quite literally very fancy associative memories.