8 ms·

Notably, humans also underperform Transformers on recall tasks. And yet we do ok on many others, even with our imperfect recall. So I hope we can identify a set

by ein0p 2y ago

Notably, humans also underperform Transformers on recall tasks. And yet we do ok on many others, even with our imperfect recall. So I hope we can identify a set of high value tasks on which these new architectures outperform Transformers and start benchmarking them on Transformers, too. Recall isn’t really “all you need” in this space, although it certainly impresses and helps to plug the capability gaps.

refulgentis 2y ago
> Notably, humans also underperform Transformers on recall tasks. Source? I don't think this is remotely true based on even the most recent benchmarks I've seen.
- ein0p 2y ago
  Can you memorize “War and Peace” in one sitting and randomly and precisely recall things from it on demand? No? Didn’t think you could. Could you perhaps translate Farsi into Hawaiian and then speak it? Can you give instant, and nearly always correct to almost any textbook question irrespective of the domain? All of that is recall, and all of that GPT4 could do for well over a year.
  - refulgentis 2y ago
    > Can you memorize “War and Peace in one sitting - neither can an LLM, unless you weaken it to "well I meant input that fits in context", not memorize > precisely recall things from it on demand - "precisely" is load-bearing here, and the question at hand. (no, LLMs don't have perfect or precise recall, even from in-context material, and it's not particularly close) > translate Farsi into Hawaiian and then speak it - 0 models, is this supposed to be a gpt-4o reference? What does this have to do with "perfect recall?" Let's pretend it does. Are you claiming that gpt-4o has a 100% success rate translating Farsi to Hawaiian? > Can you give instant, and nearly always correct to almost any textbook question - sobs in just spent 4 days benchmarking. For the record, best non-RAG I have is gpt-4o at 86% on MedQA and LegalBench. With RAG, 93%. RAG gpt-4o just barely scrapes by my cofounders USMLE scores. It's excellent. But it's not superhuman. > All of that is recall - no, translation is not recall
    - imtringued 2y ago
      You're quoting yourself. It's hard to understand which quotes are yours or not.
      - ein0p 2y ago
        Ironically, a failed recall issue. They forgot they needed to add newlines in between. :-)
    - ein0p 2y ago
      But you see, while some of those things aren’t _just_ recall, they are recall, too. Just different types of it, rather than point recall. And a modern SOTA LLM can accurately recall _vastly more_ data than any human or even any group of humans. Transformers are quite literally very fancy associative memories.
cs702 2y ago
> Notably, humans also underperform Transformers on recall tasks... Thank you. Yes, that's obviously true... but keep in mind that human beings can pull up information from notes, books, URLs, etc. and store that information in "short-term memory" at any time, as needed, for cognitive tasks. To match or exceed peak human performance, AI models must be able to do that too -- whether it's via a long context window or some other mechanism (e.g., searching on external storage) is a secondary issue. I'd say recall is a prerequisite to match peak human performance.

by ein0p 2y ago

refulgentis 2y ago

> Notably, humans also underperform Transformers on recall tasks. Source? I don't think this is remotely true based on even the most recent benchmarks I've seen.

ein0p 2y ago
Can you memorize “War and Peace” in one sitting and randomly and precisely recall things from it on demand? No? Didn’t think you could. Could you perhaps translate Farsi into Hawaiian and then speak it? Can you give instant, and nearly always correct to almost any textbook question irrespective of the domain? All of that is recall, and all of that GPT4 could do for well over a year.
- refulgentis 2y ago
  > Can you memorize “War and Peace in one sitting - neither can an LLM, unless you weaken it to "well I meant input that fits in context", not memorize > precisely recall things from it on demand - "precisely" is load-bearing here, and the question at hand. (no, LLMs don't have perfect or precise recall, even from in-context material, and it's not particularly close) > translate Farsi into Hawaiian and then speak it - 0 models, is this supposed to be a gpt-4o reference? What does this have to do with "perfect recall?" Let's pretend it does. Are you claiming that gpt-4o has a 100% success rate translating Farsi to Hawaiian? > Can you give instant, and nearly always correct to almost any textbook question - sobs in just spent 4 days benchmarking. For the record, best non-RAG I have is gpt-4o at 86% on MedQA and LegalBench. With RAG, 93%. RAG gpt-4o just barely scrapes by my cofounders USMLE scores. It's excellent. But it's not superhuman. > All of that is recall - no, translation is not recall
  - imtringued 2y ago
    You're quoting yourself. It's hard to understand which quotes are yours or not.
    - ein0p 2y ago
      Ironically, a failed recall issue. They forgot they needed to add newlines in between. :-)
  - ein0p 2y ago
    But you see, while some of those things aren’t _just_ recall, they are recall, too. Just different types of it, rather than point recall. And a modern SOTA LLM can accurately recall _vastly more_ data than any human or even any group of humans. Transformers are quite literally very fancy associative memories.

cs702 2y ago

> Notably, humans also underperform Transformers on recall tasks... Thank you. Yes, that's obviously true... but keep in mind that human beings can pull up information from notes, books, URLs, etc. and store that information in "short-term memory" at any time, as needed, for cognitive tasks. To match or exceed peak human performance, AI models must be able to do that too -- whether it's via a long context window or some other mechanism (e.g., searching on external storage) is a secondary issue. I'd say recall is a prerequisite to match peak human performance.