3 ms·

I'm going to go against the grain here and hypothesize that the computational requirements for emulating human reason are much lower than commonly claimed, and

I'm going to go against the grain here and hypothesize that the computational requirements for emulating human reason are much lower than commonly claimed, and probably within the reach of a high end GPU released in the last/next 10 years. If you look at state of the art pre-trained transformers, they gobble up unfathomable quantities of dense textual data that far exceed what even the most learned and smart humans could traverse in centuries. Yet, their results are barely intelligent. The human neural network, on the other hand, stores and is able to recall an estimated average of a few bits per waking second for a decade or two, barely a few GB of useful storage [1]. A great deal of effort and substance of the brain is dedicated to compressing the sensory input into an internal representation useful for rational though - a problem much simplified for a text-fed neural network. The ability of blind and deaf people (or even blind-deaf) for rational reasoning show these ancillary tasks are not intrinsic to human reason. So things seem to indicate that what we lack is not hardware, but the right architectures and algorithms to unlock human level reason, the exceptionally tight self training loop of humans that can make sense of the world and even push humanity's understanding further using only a few GB of compressed training data. [1] LANDAUER TK. How much do people remember ‐ some estimates of the quantity of learned information in long‐term‐memory.

sdwr 2y ago
One GPU running continuously is arguable, the 10s of thousands that go into training LLM have definitely way over shot human processing flops. Evolution definitely still has us beat on optimization-work put into thinking. Some implemented hacks: - emotions as shortcuts for reasoning - learning through imitation - self-regulation through relation
TheRoque 2y ago
> If you look at state of the art pre-trained transformers, they gobble up unfathomable quantities of dense textual data that far exceed what even the most learned and smart humans couldn't traverse in centuries. Yet, their results are barely intelligent Actually this is untrue. Yann Lecun himself said it. A kid at the age of 6 has had much much more data go through his brain (based on the visual data) than what any transformer has ingested. Moreover, words are already compressed data with a lot of lost information, nowhere near the raw experience about the world that our brains have. On top of that there's the evolutionary stuff.
- cornholio 2y ago
  > Moreover, words are already compressed data with a lot of lost information That's exactly the point, 99.9% of the information processed by the 6 year old has went into training his visual cortex, a subsystem not necessary for reasoning. This will eventually be used 8 hours a day for transforming visual representation of text on a computer screen into a compressed textual form which the inner rational loop can easily manipulate. The holy grail of AI, for anything but artistic creation, is just this inner loop.
  - TheRoque 2y ago
    This sounds like a huge oversimplification. A lot of knowledge doesn't come in the form of words, animals, early humans, kids, all don't have speech systems yet are capable of reasoning. You can't become Mozart just by reading the theory of his work, you can't become an expert at anything just by reading the theory in fact, even though it lifts you up. There's definitely things that you know and that aren't in the form of words.
    - cornholio 2y ago
      > There's definitely things that you know and that aren't in the form of words. But are they necessary for reasoning? Records of deaf-blinds with IQs a few standard deviations above average seem to suggest even highly restricted sensory inputs can allow for rational thought. It stretches credulity that you could convey more than a few bits of meaning per second to a child purely by hand gestures and touch, let alone the exabytes claimed. Of course someone like Yann Lecun will say scale is the only solution, he leads one of the juggernauts that uses scale as a competitive moat. We shouldn't expect AI researchers in positions of power to be any less vane, power hungry or human than any other humans.
      - TheRoque 2y ago
        This is on point that you mention IQ, since in IQ itself, only a single test includes words. The rest is about pattern recognition, working memory, spatial capabilities and other stuff I don't remember. So in other words, the metric you use as a benchmark for good reasoning barely measures the literacy

sdwr 2y ago

One GPU running continuously is arguable, the 10s of thousands that go into training LLM have definitely way over shot human processing flops. Evolution definitely still has us beat on optimization-work put into thinking. Some implemented hacks: - emotions as shortcuts for reasoning - learning through imitation - self-regulation through relation

TheRoque 2y ago

> If you look at state of the art pre-trained transformers, they gobble up unfathomable quantities of dense textual data that far exceed what even the most learned and smart humans couldn't traverse in centuries. Yet, their results are barely intelligent Actually this is untrue. Yann Lecun himself said it. A kid at the age of 6 has had much much more data go through his brain (based on the visual data) than what any transformer has ingested. Moreover, words are already compressed data with a lot of lost information, nowhere near the raw experience about the world that our brains have. On top of that there's the evolutionary stuff.

cornholio 2y ago
> Moreover, words are already compressed data with a lot of lost information That's exactly the point, 99.9% of the information processed by the 6 year old has went into training his visual cortex, a subsystem not necessary for reasoning. This will eventually be used 8 hours a day for transforming visual representation of text on a computer screen into a compressed textual form which the inner rational loop can easily manipulate. The holy grail of AI, for anything but artistic creation, is just this inner loop.
- TheRoque 2y ago
  This sounds like a huge oversimplification. A lot of knowledge doesn't come in the form of words, animals, early humans, kids, all don't have speech systems yet are capable of reasoning. You can't become Mozart just by reading the theory of his work, you can't become an expert at anything just by reading the theory in fact, even though it lifts you up. There's definitely things that you know and that aren't in the form of words.
  - cornholio 2y ago
    > There's definitely things that you know and that aren't in the form of words. But are they necessary for reasoning? Records of deaf-blinds with IQs a few standard deviations above average seem to suggest even highly restricted sensory inputs can allow for rational thought. It stretches credulity that you could convey more than a few bits of meaning per second to a child purely by hand gestures and touch, let alone the exabytes claimed. Of course someone like Yann Lecun will say scale is the only solution, he leads one of the juggernauts that uses scale as a competitive moat. We shouldn't expect AI researchers in positions of power to be any less vane, power hungry or human than any other humans.
    - TheRoque 2y ago
      This is on point that you mention IQ, since in IQ itself, only a single test includes words. The rest is about pattern recognition, working memory, spatial capabilities and other stuff I don't remember. So in other words, the metric you use as a benchmark for good reasoning barely measures the literacy