6 ms·

We are throwing unheared amounts of money in AI and unseen compute. Progress is huge and fast and we barely started. If this progress and focus and resources d

We are throwing unheared amounts of money in AI and unseen compute. Progress is huge and fast and we barely started. If this progress and focus and resources doesn't lead to AI despite us already seeing a system which was unimaginable 6 years ago, we will never see AGI. And if you look at Boston Dynamics, Unitree and Generalist's progress on robotics, thats also CRAZY.

mort96 6mo ago
If I'm reading you right, your opinion is essentially: "If building bigger and bigger statistical next word predictors won't lead to artificial general intelligence, we will never see artificial general intelligence" I don't know, maybe AGI is possible but there's more to intelligence than statistical next word prediction?
- AntiUSAbah 6mo ago
  Its not a statistical next word predictor. The 'predicting the next word' is the learning mechanism of the LLM which leads to a latent space which can encode higher level concepts. Basically a LLM 'understands' that much as efficient as it has to be to be able to respond in a reasonable way. A LLM doesn't predict german text or chinese language. It predicts the concept and than has a language layer outputting tokens. And its not just LLMs which are progressing fast, voice synt and voice understanding jumped significantly, motion detection, skeletion movement, virtual world generation (see nvidias way of generating virutal worlds for their car training), protein folding etc.
  - mort96 6mo ago
    I'm sorry but the input to a model is a sequence of tokens and the output is a probability distribution of what's the most likely next token. It's a very very very fancy next token predictor but that is fundamentally what it is. I'm making the argument that this paradigm might not give rise to a general intelligence no matter how much you scale it.
    - CamperBob2 6mo ago
      It's a very very very fancy next token predictor Yes, and unless you are prepared to rebut the argument with evidence of the supernatural, that's all there is, period. That's all we are. So tired of the thought-terminating "stochastic parrot" argument.
      - mort96 6mo ago
        I'm not sure why you think you know the human brain works through predicting the next token. It's not supernatural, I believe that an artificial intelligence is possible because I believe human intelligence is just a clever arrangement of matter performing computation, but I would never be presumptuous enough to claim to know exactly how that mechanism works. My opinion is that human intelligence might be what's essentially a fancy next token predictor, or it might work in some completely different way, I don't know. Your claim is that human intelligence is a next token predictor. It seems like the burden on proof is on you.
        dpark 6mo ago
        > Your claim is that human intelligence is a next token predictor. Literally it is, at least in many of its forms. You accepted CamperBob2’s text as input and then you generated text as output. Unless you are positing that this behavior cannot prove your own general intelligence, it seems plain that “next token generator” is sufficient for AGI. (Whether the current LLM architecture is sufficient is a slightly different question.)
        mort96 6mo ago
        Before I start typing, I think abstractly about the topic and decide on what I shall write in response. Due to the linear nature of time, typing necessarily happens one word at a time, but I am never producing a probability distribution of words (at least not in a way that my conscious self can determine), I consider an entire idea and then decide what tokens to enter into the computer in order to communicate the idea to you. And while I am typing, and while I am thinking before I type, I experience an array of non-textual sensory input, and my whole experience of self is to a significant extent non-lingual. Sometimes, I experience an inner monologue, sometimes I think thoughts which aren't expressed in language such as the structure of the data flow in a computer program, sometimes I don't think and just experience feelings like a kiss or the sun on my skin or the euphoria of a piece of music which hits just right. These experiences shape who I am and how I think. When I solve difficult programming problems or other difficult problems, I build abstract structures in my mind which represents the relevant information and consider things like how data flows, which parts impact which other parts, what the constraints are, etc. without language coming in to play at all. This process seems completely detached from words. In contrast, for a language model, there is no thinking outside of producing words. It seems self-evident to me that at least parts of the human experience fundamentally can not be reduced to next token prediction. Further, it seems plausible to me that some of these aspects may be necessary for what we consider general intelligence. Therefore, my position is: it is plausible that next token prediction won't give rise to general intelligence, and I do not find your argument convincing.
  - somewhereoutth 6mo ago
    LLM proponents believe that these higher level encodings in latent space do in fact match the real world concepts described by our language(s). However, a much simpler explanation for what we see with LLMs is that instead the higher level encodings in latent space match only the patterns of our language(s), and no deeper encoding/understanding is present. It's Plato's Cave - the shadows on the wall are all an LLM ever sees, and somehow it is expected to derive the real reality behind them.
    - AntiUSAbah 6mo ago
      Could be, yes for sure but I think it would be very naive in the current state of progress we are in, to down play what progress is happening. At least Mythos model with its 10 Trillion parameter might indicate that the scaling law is valid. Its a little bit unfortunate that we still don't know that much more about that model.
  - turtlesdown11 6mo ago
    > Its not a statistical next word predictor. it absolutely is a next word predictor
bmitc 6mo ago
Same thing happened with self-driving cars. Oh and cryptocurrencies.
- AntiUSAbah 6mo ago
  Self-driving had never the amount of compute, research adoption and money than what the current overall AI has. Its not comparable. Crypto was flawed from the beginning and lots of people didn't understood it properly. Not even that a blockchain can't secure a transaction from something outside of a blockchain.
  - bigfishrunning 6mo ago
    The LLMs are flawed, and lots of people don't understand them properly.
    - AntiUSAbah 6mo ago
      People are researching how to make LLMs more stable and from a statistic point of view, we already now down to 10% (progress is made here). LLMs don't have to be perfect, they just need to be as good as humans and cheaper or easier to manage.
  - freejazz 6mo ago
    > Self-driving had never the amount of compute, research adoption and money than what the current overall AI has. And yet they don't do really good jobs with pretty much anything, save for software development, to which people still seem pretty split as far as it being a helpful thing. That's before we even factor in the cost.
    - AntiUSAbah 6mo ago
      I find them very helpful. I use gemini regularly for multiply things. I also believe that whatever code researchers and other non software engineers wrote before coding agents, were similiar shitty but took them a lot longer to write. Like do you know how many researchers need to do some data analysis and hack around code because they never learned programming? So so many. If they know how to verify their data (which they needed to know before already), a LLM helps them already. There is also plenty of other code were perfection doesn't matter. Non SaaS software exists. For security experts, we just saw whats happening. The curl inventor mentioned it online that the newest AI reports for Security issues are real and the amount of security gaps found are real and a lot of work. Image generation is very good and you can see it today already everywere. From cheap restaurants using it, to invitations, whatsapp messages, social media, advertising. I have a work collegue, who is in it for 6 years and he studied, he is so underqualified if you give me his salary as tokens today, i wouldn't think for a second to replace him.
benterix 6mo ago
Not sure if you're being sincere or sarcastic but some of us have lived through several AI winters now. And the fact that such a phenomenon exists is because of this terrible amount of hype the topic gets whenever any progress is made.
- AntiUSAbah 6mo ago
  Which ones? At least in the last 4 years, there was no AI winter.
  - sumeno 6mo ago
    History started well before 4 years ago
    - AntiUSAbah 6mo ago
      Yeah but this AI wave has nothing to do how we came to AI winter in the 70s or 80s. The necessary amount of Compute, interconnect (internet), money, researcher etc. wasn't available at that time. and we did not invest the most amount of money and compute and brain power as we are doing right now. This is unseen.
      - toyg 6mo ago
        Ah, the youth... "The new economy" also didn't have anything to do with the previous one. Turns out that it crashed just as well.
        AntiUSAbah 6mo ago
        I'm not an economist but at least as an european person, I currently do see a huge restructuring going on. A shift away from the USA to China. But I never voiced an opinion about that. I do follow ML/AI/AGI though for a decade by now and read a lot about Neuronal networks, LLMs, etc. in a broad spectrum. My prediction regarding Crypto/blockchain was true too. We will see how it plays out. I'm open for both, but I think it would be naive to ignore whats going on and its way to soon to assume there is a AI winter coming soon. We sitll want to see what Mythos can do and a distilled version of it.
linhns 6mo ago
> And if you look at Boston Dynamics, Unitree and Generalist's progress on robotics Their progress is almost nought. Humanoids are stupid creations that are not good at anything in the real world. I'll give it to the machine dogs, at least they can reach corners we cannot.
- AntiUSAbah 6mo ago
  I found there demonstration at the CES this year very spectacular: https://www.youtube.com/watch?v=YIhzUnvi7Fw https://www.youtube.com/watch?v=YIhzUnvi7Fw I can also recommend looking at Generalist: https://www.youtube.com/@Generalist_AI https://www.youtube.com/@Generalist_AI
- fragmede 6mo ago
  > Their progress is almost nought. How can you say the advancements since Honda's asimo robot amount to "almost naought"?
turtlesdown11 6mo ago
> Progress is huge and fast is it? we're currently scaled on data input and LLMs in general, the only thing making them advance at all right now is adding processing power

mort96 6mo ago

If I'm reading you right, your opinion is essentially: "If building bigger and bigger statistical next word predictors won't lead to artificial general intelligence, we will never see artificial general intelligence" I don't know, maybe AGI is possible but there's more to intelligence than statistical next word prediction?

AntiUSAbah 6mo ago
Its not a statistical next word predictor. The 'predicting the next word' is the learning mechanism of the LLM which leads to a latent space which can encode higher level concepts. Basically a LLM 'understands' that much as efficient as it has to be to be able to respond in a reasonable way. A LLM doesn't predict german text or chinese language. It predicts the concept and than has a language layer outputting tokens. And its not just LLMs which are progressing fast, voice synt and voice understanding jumped significantly, motion detection, skeletion movement, virtual world generation (see nvidias way of generating virutal worlds for their car training), protein folding etc.
- mort96 6mo ago
  I'm sorry but the input to a model is a sequence of tokens and the output is a probability distribution of what's the most likely next token. It's a very very very fancy next token predictor but that is fundamentally what it is. I'm making the argument that this paradigm might not give rise to a general intelligence no matter how much you scale it.
  - CamperBob2 6mo ago
    It's a very very very fancy next token predictor Yes, and unless you are prepared to rebut the argument with evidence of the supernatural, that's all there is, period. That's all we are. So tired of the thought-terminating "stochastic parrot" argument.
    - mort96 6mo ago
      I'm not sure why you think you know the human brain works through predicting the next token. It's not supernatural, I believe that an artificial intelligence is possible because I believe human intelligence is just a clever arrangement of matter performing computation, but I would never be presumptuous enough to claim to know exactly how that mechanism works. My opinion is that human intelligence might be what's essentially a fancy next token predictor, or it might work in some completely different way, I don't know. Your claim is that human intelligence is a next token predictor. It seems like the burden on proof is on you.
      - dpark 6mo ago
        > Your claim is that human intelligence is a next token predictor. Literally it is, at least in many of its forms. You accepted CamperBob2’s text as input and then you generated text as output. Unless you are positing that this behavior cannot prove your own general intelligence, it seems plain that “next token generator” is sufficient for AGI. (Whether the current LLM architecture is sufficient is a slightly different question.)
        mort96 6mo ago
        Before I start typing, I think abstractly about the topic and decide on what I shall write in response. Due to the linear nature of time, typing necessarily happens one word at a time, but I am never producing a probability distribution of words (at least not in a way that my conscious self can determine), I consider an entire idea and then decide what tokens to enter into the computer in order to communicate the idea to you. And while I am typing, and while I am thinking before I type, I experience an array of non-textual sensory input, and my whole experience of self is to a significant extent non-lingual. Sometimes, I experience an inner monologue, sometimes I think thoughts which aren't expressed in language such as the structure of the data flow in a computer program, sometimes I don't think and just experience feelings like a kiss or the sun on my skin or the euphoria of a piece of music which hits just right. These experiences shape who I am and how I think. When I solve difficult programming problems or other difficult problems, I build abstract structures in my mind which represents the relevant information and consider things like how data flows, which parts impact which other parts, what the constraints are, etc. without language coming in to play at all. This process seems completely detached from words. In contrast, for a language model, there is no thinking outside of producing words. It seems self-evident to me that at least parts of the human experience fundamentally can not be reduced to next token prediction. Further, it seems plausible to me that some of these aspects may be necessary for what we consider general intelligence. Therefore, my position is: it is plausible that next token prediction won't give rise to general intelligence, and I do not find your argument convincing.
- somewhereoutth 6mo ago
  LLM proponents believe that these higher level encodings in latent space do in fact match the real world concepts described by our language(s). However, a much simpler explanation for what we see with LLMs is that instead the higher level encodings in latent space match only the patterns of our language(s), and no deeper encoding/understanding is present. It's Plato's Cave - the shadows on the wall are all an LLM ever sees, and somehow it is expected to derive the real reality behind them.
  - AntiUSAbah 6mo ago
    Could be, yes for sure but I think it would be very naive in the current state of progress we are in, to down play what progress is happening. At least Mythos model with its 10 Trillion parameter might indicate that the scaling law is valid. Its a little bit unfortunate that we still don't know that much more about that model.
- turtlesdown11 6mo ago
  > Its not a statistical next word predictor. it absolutely is a next word predictor

bmitc 6mo ago

Same thing happened with self-driving cars. Oh and cryptocurrencies.

AntiUSAbah 6mo ago
Self-driving had never the amount of compute, research adoption and money than what the current overall AI has. Its not comparable. Crypto was flawed from the beginning and lots of people didn't understood it properly. Not even that a blockchain can't secure a transaction from something outside of a blockchain.
- bigfishrunning 6mo ago
  The LLMs are flawed, and lots of people don't understand them properly.
  - AntiUSAbah 6mo ago
    People are researching how to make LLMs more stable and from a statistic point of view, we already now down to 10% (progress is made here). LLMs don't have to be perfect, they just need to be as good as humans and cheaper or easier to manage.
- freejazz 6mo ago
  > Self-driving had never the amount of compute, research adoption and money than what the current overall AI has. And yet they don't do really good jobs with pretty much anything, save for software development, to which people still seem pretty split as far as it being a helpful thing. That's before we even factor in the cost.
  - AntiUSAbah 6mo ago
    I find them very helpful. I use gemini regularly for multiply things. I also believe that whatever code researchers and other non software engineers wrote before coding agents, were similiar shitty but took them a lot longer to write. Like do you know how many researchers need to do some data analysis and hack around code because they never learned programming? So so many. If they know how to verify their data (which they needed to know before already), a LLM helps them already. There is also plenty of other code were perfection doesn't matter. Non SaaS software exists. For security experts, we just saw whats happening. The curl inventor mentioned it online that the newest AI reports for Security issues are real and the amount of security gaps found are real and a lot of work. Image generation is very good and you can see it today already everywere. From cheap restaurants using it, to invitations, whatsapp messages, social media, advertising. I have a work collegue, who is in it for 6 years and he studied, he is so underqualified if you give me his salary as tokens today, i wouldn't think for a second to replace him.

benterix 6mo ago

Not sure if you're being sincere or sarcastic but some of us have lived through several AI winters now. And the fact that such a phenomenon exists is because of this terrible amount of hype the topic gets whenever any progress is made.

AntiUSAbah 6mo ago
Which ones? At least in the last 4 years, there was no AI winter.
- sumeno 6mo ago
  History started well before 4 years ago
  - AntiUSAbah 6mo ago
    Yeah but this AI wave has nothing to do how we came to AI winter in the 70s or 80s. The necessary amount of Compute, interconnect (internet), money, researcher etc. wasn't available at that time. and we did not invest the most amount of money and compute and brain power as we are doing right now. This is unseen.
    - toyg 6mo ago
      Ah, the youth... "The new economy" also didn't have anything to do with the previous one. Turns out that it crashed just as well.
      - AntiUSAbah 6mo ago
        I'm not an economist but at least as an european person, I currently do see a huge restructuring going on. A shift away from the USA to China. But I never voiced an opinion about that. I do follow ML/AI/AGI though for a decade by now and read a lot about Neuronal networks, LLMs, etc. in a broad spectrum. My prediction regarding Crypto/blockchain was true too. We will see how it plays out. I'm open for both, but I think it would be naive to ignore whats going on and its way to soon to assume there is a AI winter coming soon. We sitll want to see what Mythos can do and a distilled version of it.

linhns 6mo ago

> And if you look at Boston Dynamics, Unitree and Generalist's progress on robotics Their progress is almost nought. Humanoids are stupid creations that are not good at anything in the real world. I'll give it to the machine dogs, at least they can reach corners we cannot.

AntiUSAbah 6mo ago
I found there demonstration at the CES this year very spectacular: https://www.youtube.com/watch?v=YIhzUnvi7Fw https://www.youtube.com/watch?v=YIhzUnvi7Fw I can also recommend looking at Generalist: https://www.youtube.com/@Generalist_AI https://www.youtube.com/@Generalist_AI
fragmede 6mo ago
> Their progress is almost nought. How can you say the advancements since Honda's asimo robot amount to "almost naought"?

turtlesdown11 6mo ago

> Progress is huge and fast is it? we're currently scaled on data input and LLMs in general, the only thing making them advance at all right now is adding processing power