7 ms·

I find their critique compelling, particularly their emphasis on the disconnect between CoT’s algorithmic mimicry and true cognitive exploration. The authors il

by drcwpl 2y ago

I find their critique compelling, particularly their emphasis on the disconnect between CoT’s algorithmic mimicry and true cognitive exploration. The authors illustrate this with examples from advanced mathematics, such as the "windmill problem" from the International Mathematics Olympiad, a puzzle whose solution eludes brute-force sequential thinking. These cases underscore the limits of a framework that relies on static datasets and rigid generative processes. CoT, as they demonstrate, falters not because it cannot generate solutions, but because it cannot conceive of them in ways that mirror human ingenuity. As they say - "Superintelligence isn't about discovering new things; it's about discovering new ways to discover."

seagullz 2y ago
And then other problems would perhaps turn up down the track that would call for "discovering new ways to discover new ways of discovery" and so on.
dartos 2y ago
> "Superintelligence isn't about discovering new things; it's about discovering new ways to discover." Wow I love that quote.
- leobg 2y ago
  That’s meta. Literally. Edit: Sorry. This was based on the false assumption that this was research by Meta, Inc..
WillieCubed 2y ago
I love the quote you mentioned at the end. Do you remember the original source?
- deleted 2y ago
  [deleted]
- fragmede 2y ago
  https://x.com/nathanthinks/status/1877510438621163987 https://x.com/nathanthinks/status/1877510438621163987
KaoruAoiShiho 2y ago
Just train it on meta reasoning, ie train it on people discovering ways to discover. It's not really a big problem, just generate the dataset and have at it.
- derefr 2y ago
  This doesn't give you the ability to process ideas through the derived new insights, any more than loading the contents of a VLSI program into regular RAM gives you an FPGA. The linear-algebra primitives used in LLM inference, fundamentally do not have the power to allow an LLM to "emulate" its own internals (i.e. to have the [static!] weights + [runtime-mutable] context, together encode [runtime-mutable] virtual weights, that the same host context can be passed through.) You need host support for that.
  - lxgr 2y ago
    > The linear-algebra primitives used in LLM inference, fundamentally do not have the power to allow an LLM to "emulate" its own internals […] You need host support for that. Neither do biological brains (explicitly), yet we can hypothesize just fine.
    - derefr 2y ago
      You're conflating two steps: 1. hypothesizing — coming up with a novel insight at runtime, that uncovers new parts of the state space the model doesn't currently reach 2. syllogizing — using an insight you've derived at runtime, to reach the new parts of the state space LLMs can do 1, but not 2. (Try it for yourself: get an LLM to prove a trivial novel mathematical theorem [or just describe the theorem to it yourself]; and then ask it to use the theorem to solve a problem. It won't be able to do it. It "understands" the theorem as data; but it doesn't have weights shaped like an emulator that can execute the theorem-modelled-as-data against the context. And, as far as I understand them, current Transformer-ish models cannot "learn" such an emulator as a feature. You need a slightly different architecture for that.) And actually, humans can't really do 2 either! That is: humans can't immediately make use of entirely-novel insights that weren't "trained in", but only just came to them, any more than LLMs can! Instead, for humans, the process we go through is either: • come up with the insight; sleep on it (i.e. do incremental training, converting the data into new weights); use the insight • build up 99% of the weights required for the insight "in the background" over days/months/years without realizing it; make the final single connection to "unlock" the insight; immediately use the insight LLMs don't get to do either of these things. LLMs don't do "memory consolidation"; there is no gradual online/semi-online conversion of "experiences" into weights, i.e. reifying the "code stored as data" into becoming "code" that can be executed as part of the model. With (current) LLMs, there's only the entirely-offline training/fine-tuning/RLHF — at much greater expense and requiring much greater hardware resources than inference does — to produce a new iteration of the model. That's why we're (currently) stuck in a paradigm of throwing prompts at ever-larger GPT base models — rather than just having an arbitrary stateful base-model that you "install" onto a device like you'd install an RDBMS, and then have it "learn on the job" from there.
TaurenHunter 2y ago
Thank you for mentioning the windmill problem. Great insights! https://www.3blue1brown.com/lessons/windmills https://www.3blue1brown.com/lessons/windmills

by drcwpl 2y ago

seagullz 2y ago

And then other problems would perhaps turn up down the track that would call for "discovering new ways to discover new ways of discovery" and so on.

dartos 2y ago

> "Superintelligence isn't about discovering new things; it's about discovering new ways to discover." Wow I love that quote.

leobg 2y ago
That’s meta. Literally. Edit: Sorry. This was based on the false assumption that this was research by Meta, Inc..

WillieCubed 2y ago

I love the quote you mentioned at the end. Do you remember the original source?

deleted 2y ago
[deleted]
fragmede 2y ago
https://x.com/nathanthinks/status/1877510438621163987 https://x.com/nathanthinks/status/1877510438621163987

KaoruAoiShiho 2y ago

Just train it on meta reasoning, ie train it on people discovering ways to discover. It's not really a big problem, just generate the dataset and have at it.

derefr 2y ago
This doesn't give you the ability to process ideas through the derived new insights, any more than loading the contents of a VLSI program into regular RAM gives you an FPGA. The linear-algebra primitives used in LLM inference, fundamentally do not have the power to allow an LLM to "emulate" its own internals (i.e. to have the [static!] weights + [runtime-mutable] context, together encode [runtime-mutable] virtual weights, that the same host context can be passed through.) You need host support for that.
- lxgr 2y ago
  > The linear-algebra primitives used in LLM inference, fundamentally do not have the power to allow an LLM to "emulate" its own internals […] You need host support for that. Neither do biological brains (explicitly), yet we can hypothesize just fine.
  - derefr 2y ago
    You're conflating two steps: 1. hypothesizing — coming up with a novel insight at runtime, that uncovers new parts of the state space the model doesn't currently reach 2. syllogizing — using an insight you've derived at runtime, to reach the new parts of the state space LLMs can do 1, but not 2. (Try it for yourself: get an LLM to prove a trivial novel mathematical theorem [or just describe the theorem to it yourself]; and then ask it to use the theorem to solve a problem. It won't be able to do it. It "understands" the theorem as data; but it doesn't have weights shaped like an emulator that can execute the theorem-modelled-as-data against the context. And, as far as I understand them, current Transformer-ish models cannot "learn" such an emulator as a feature. You need a slightly different architecture for that.) And actually, humans can't really do 2 either! That is: humans can't immediately make use of entirely-novel insights that weren't "trained in", but only just came to them, any more than LLMs can! Instead, for humans, the process we go through is either: • come up with the insight; sleep on it (i.e. do incremental training, converting the data into new weights); use the insight • build up 99% of the weights required for the insight "in the background" over days/months/years without realizing it; make the final single connection to "unlock" the insight; immediately use the insight LLMs don't get to do either of these things. LLMs don't do "memory consolidation"; there is no gradual online/semi-online conversion of "experiences" into weights, i.e. reifying the "code stored as data" into becoming "code" that can be executed as part of the model. With (current) LLMs, there's only the entirely-offline training/fine-tuning/RLHF — at much greater expense and requiring much greater hardware resources than inference does — to produce a new iteration of the model. That's why we're (currently) stuck in a paradigm of throwing prompts at ever-larger GPT base models — rather than just having an arbitrary stateful base-model that you "install" onto a device like you'd install an RDBMS, and then have it "learn on the job" from there.

TaurenHunter 2y ago

Thank you for mentioning the windmill problem. Great insights! https://www.3blue1brown.com/lessons/windmills https://www.3blue1brown.com/lessons/windmills