5 ms·

>What do you think the last few years have been all about? Next token language-based predictors with no more intelligence than brute force GIGO which parrot ex

>What do you think the last few years have been all about? Next token language-based predictors with no more intelligence than brute force GIGO which parrot existing human intelligence captured as text/audio and fed in the form of input data. 4o agrees: "What you are describing is a language model or next-token predictor that operates solely as a computational system without inherent intelligence or understanding. The phrase captures the essence of generative AI models, like GPT, which rely on statistical and probabilistic methods to predict the next piece of text based on patterns in the data they’ve been trained on"

thrwthsnw 2y ago
Everything you said is parroting data you’ve trained on, two thirds of it is actual copy paste
- coldtea 2y ago
  >Everything you said is parroting data you’ve trained on "Just like" an LLM, yeah sure... Like how the brain was "just like" a hydraulic system (early industrial era), like a clockwork with gears and differentiation (mechanical engineering), "just like" an electric circuit (Edison's time), "just like" a computer CPU (21st century), and so on... You're just assuming what you should prove
- mrbungie 2y ago
  He probably didn't need petabytes of reddit posts and millions of gpu-hours to parrot that though. I still don't buy the "we do the same as LLMs" discourse. Of course one could hypothesize the human brain language center may have some similarities to LLMs, but the differences in resource usage and how those resources are used to train humans and LLMs are remarkable and may indicate otherwise.
  - shwouchk 2y ago
    Not text, he had petabytes of video, audio, and other sensory inputs. Heck, a baby sees petabytes of video before first word is spoken And he probably cant quote Shakespeare as well ;)
    - Jensson 2y ago
      > he had petabytes of video, audio, and other sensory inputs He didn't parrot a video or sensory inputs though.
      - shwouchk 2y ago
        No, they don’t - they don’t have the hardware, yet. But they do parrot sensory output to eg muscles that induce the expected video sensory inputs in response, in a way that mimics the video input of “other people doing things”.
    - mrbungie 2y ago
      And yet with multiple OoM more data he still didn't cost millions of dollars to be trained nor multiple lifetimes in gpu-hours. He probably didn't even register all the petabytes passing through all his "sensors", those are some characteristics that we are not even near understanding and much less replicating. Whatever is happening in the brain is more complex as the perf/cost ratio is stupidly better for humans for a lot of tasks in both training and inference*. *when considering all modalities, o3 can't even do the ARC AGI in vision mode but rather just json representations. So much for omni.
    - coldtea 2y ago
      >Not text, he had petabytes of video, audio, and other sensory inputs. Heck, a baby sees petabytes of video before first word is spoken A 2-3 year old baby could speak in a rural village in 1800, having just seen its cradle (for the first month/s), and its parents' hut for some more months, and maybe parts of the village afterwards. Hardly "petabytes of training video" to write home about.
      - shwouchk 2y ago
        you are funny. Clearly your expertise with babies comes from reading books about history or science, rather than ever having interacted with one… What resolution of screen do you think you would need to not distinguish from reality? For me personally i very conservatively estimate it to be on above OOM of 10 4k screens by 10, meaning 100k screens. If a typical 2h 4k is ~50gb uncompressed, that gives us about half a petabyte per 24h (even with eyes closed). Just raw unlabeled vision data. Probably a baby has a significantly lower resolution, but then again what is the resolution from the skin and other organs? So yes, petabytes of data within the first days of existence - well, likely before even being born since baby can hear inside the uterus, for example. And very high signal data, as you’ve stated yourself (nothing to write home about) mainly seeing mom and dad, as well as from a feedback loop POV - a baby never tells you it is hungry subtly.
  - deleted 2y ago
    [deleted]
Eisenstein 2y ago
You have described something but you haven't explained why the description of the thing defines its capability. This is a tautology, or possibly a begging of the question, which takes as true the premise of something (that token based language predictors cannot be intelligent) and then uses that premise to prove an unproven point (that language models cannot achieve intelligence). You did nothing at all to demonstrate why you cannot produce an intelligent system from a next token language based predictor. What GPT says about this is completely irrelevant.
- coldtea 2y ago
  >You did nothing at all to demonstrate why you cannot produce an intelligent system from a next token language based predictor Sorry, but the burden of proof is on your side... The intelligence is in the corpus the LLM was fed with. Using statistics to pick from it and re-arrange it gives new intelligent results because the information was already produced by intelligent beings. If somebody gives you an excerpt of a book, it doesn't mean they have the intelligence of the author - even if you have taught them a mechanical statistical method to give back a section matching a query you make. Kids learn to speak and understand language at 3-4 years old (among tons of other concepts), and can reason by themselves in a few years with less than 1 billionth the input... >What GPT says about this is completely irrelevant. On the contrary, it's using its very real intelligence, about to reach singularity any time now, and this is its verdict! Why would you say it's irrelevant? That would be as if it merely statistically parroted combinations of its training data unconnected to any reasoning (except of that the human creators of the data used to create them) or objective reality...
  - Eisenstein 2y ago
    Let's pretend it is 1940 Person 1: rockets could be a method of putting things into Earth orbit Person 2: rockets cannot get things into orbit because they use a chemical reaction which causes an equal and opposite force reaction to produce thrust' Does person 1 have the burden of proof that rockets can be used to put things in orbit? Sure, but that doesn't make the reasoning used by person 2 valid to explain why person 1 is wrong. BTW thanks for adding an entire chapter to your comment in edit so it looks like I am ignoring most of it. What I replied to was one sentence that said 'the burden of proof is on you'. Though it really doesn't make much difference because you are doing the same thing but more verbose this time. None of the things you mentioned preclude intelligence. You are telling us again how it operates but not why that operation is restrictive in producing an intelligent output. There is no law that saws that intelligence requires anything but a large amount of data and computation. If you can show why these things are not sufficient, I am eager to read about it. A logical explanation would be great, step by step please, without making any grand unproven assumptions. In response to the person below... again, whether or not person 1 is right or wrong does not make person 2's argument valid.
resters 2y ago
This comment isn't false but it's very naive.
zmgsabst 2y ago
o1 points out this is mostly about “if submarines swim”. https://chatgpt.com/share/6768c920-4454-8000-bf73-0f86e92996e4 https://chatgpt.com/share/6768c920-4454-8000-bf73-0f86e92996...
ComplexSystems 2y ago
What do you think "AGI" is supposed to be?

thrwthsnw 2y ago

Everything you said is parroting data you’ve trained on, two thirds of it is actual copy paste

coldtea 2y ago
>Everything you said is parroting data you’ve trained on "Just like" an LLM, yeah sure... Like how the brain was "just like" a hydraulic system (early industrial era), like a clockwork with gears and differentiation (mechanical engineering), "just like" an electric circuit (Edison's time), "just like" a computer CPU (21st century), and so on... You're just assuming what you should prove
mrbungie 2y ago
He probably didn't need petabytes of reddit posts and millions of gpu-hours to parrot that though. I still don't buy the "we do the same as LLMs" discourse. Of course one could hypothesize the human brain language center may have some similarities to LLMs, but the differences in resource usage and how those resources are used to train humans and LLMs are remarkable and may indicate otherwise.
- shwouchk 2y ago
  Not text, he had petabytes of video, audio, and other sensory inputs. Heck, a baby sees petabytes of video before first word is spoken And he probably cant quote Shakespeare as well ;)
  - Jensson 2y ago
    > he had petabytes of video, audio, and other sensory inputs He didn't parrot a video or sensory inputs though.
    - shwouchk 2y ago
      No, they don’t - they don’t have the hardware, yet. But they do parrot sensory output to eg muscles that induce the expected video sensory inputs in response, in a way that mimics the video input of “other people doing things”.
  - mrbungie 2y ago
    And yet with multiple OoM more data he still didn't cost millions of dollars to be trained nor multiple lifetimes in gpu-hours. He probably didn't even register all the petabytes passing through all his "sensors", those are some characteristics that we are not even near understanding and much less replicating. Whatever is happening in the brain is more complex as the perf/cost ratio is stupidly better for humans for a lot of tasks in both training and inference*. *when considering all modalities, o3 can't even do the ARC AGI in vision mode but rather just json representations. So much for omni.
  - coldtea 2y ago
    >Not text, he had petabytes of video, audio, and other sensory inputs. Heck, a baby sees petabytes of video before first word is spoken A 2-3 year old baby could speak in a rural village in 1800, having just seen its cradle (for the first month/s), and its parents' hut for some more months, and maybe parts of the village afterwards. Hardly "petabytes of training video" to write home about.
    - shwouchk 2y ago
      you are funny. Clearly your expertise with babies comes from reading books about history or science, rather than ever having interacted with one… What resolution of screen do you think you would need to not distinguish from reality? For me personally i very conservatively estimate it to be on above OOM of 10 4k screens by 10, meaning 100k screens. If a typical 2h 4k is ~50gb uncompressed, that gives us about half a petabyte per 24h (even with eyes closed). Just raw unlabeled vision data. Probably a baby has a significantly lower resolution, but then again what is the resolution from the skin and other organs? So yes, petabytes of data within the first days of existence - well, likely before even being born since baby can hear inside the uterus, for example. And very high signal data, as you’ve stated yourself (nothing to write home about) mainly seeing mom and dad, as well as from a feedback loop POV - a baby never tells you it is hungry subtly.
- deleted 2y ago
  [deleted]

Eisenstein 2y ago

You have described something but you haven't explained why the description of the thing defines its capability. This is a tautology, or possibly a begging of the question, which takes as true the premise of something (that token based language predictors cannot be intelligent) and then uses that premise to prove an unproven point (that language models cannot achieve intelligence). You did nothing at all to demonstrate why you cannot produce an intelligent system from a next token language based predictor. What GPT says about this is completely irrelevant.

coldtea 2y ago
>You did nothing at all to demonstrate why you cannot produce an intelligent system from a next token language based predictor Sorry, but the burden of proof is on your side... The intelligence is in the corpus the LLM was fed with. Using statistics to pick from it and re-arrange it gives new intelligent results because the information was already produced by intelligent beings. If somebody gives you an excerpt of a book, it doesn't mean they have the intelligence of the author - even if you have taught them a mechanical statistical method to give back a section matching a query you make. Kids learn to speak and understand language at 3-4 years old (among tons of other concepts), and can reason by themselves in a few years with less than 1 billionth the input... >What GPT says about this is completely irrelevant. On the contrary, it's using its very real intelligence, about to reach singularity any time now, and this is its verdict! Why would you say it's irrelevant? That would be as if it merely statistically parroted combinations of its training data unconnected to any reasoning (except of that the human creators of the data used to create them) or objective reality...
- Eisenstein 2y ago
  Let's pretend it is 1940 Person 1: rockets could be a method of putting things into Earth orbit Person 2: rockets cannot get things into orbit because they use a chemical reaction which causes an equal and opposite force reaction to produce thrust' Does person 1 have the burden of proof that rockets can be used to put things in orbit? Sure, but that doesn't make the reasoning used by person 2 valid to explain why person 1 is wrong. BTW thanks for adding an entire chapter to your comment in edit so it looks like I am ignoring most of it. What I replied to was one sentence that said 'the burden of proof is on you'. Though it really doesn't make much difference because you are doing the same thing but more verbose this time. None of the things you mentioned preclude intelligence. You are telling us again how it operates but not why that operation is restrictive in producing an intelligent output. There is no law that saws that intelligence requires anything but a large amount of data and computation. If you can show why these things are not sufficient, I am eager to read about it. A logical explanation would be great, step by step please, without making any grand unproven assumptions. In response to the person below... again, whether or not person 1 is right or wrong does not make person 2's argument valid.

resters 2y ago

This comment isn't false but it's very naive.

zmgsabst 2y ago

o1 points out this is mostly about “if submarines swim”. https://chatgpt.com/share/6768c920-4454-8000-bf73-0f86e92996e4 https://chatgpt.com/share/6768c920-4454-8000-bf73-0f86e92996...

ComplexSystems 2y ago

What do you think "AGI" is supposed to be?