7 ms·

There is a vast gulf between theoretically possible and technologically feasible. If you can’t provide a realistic path to achieve something, you’re asking peo

by davemp 3mo ago

There is a vast gulf between theoretically possible and technologically feasible. If you can’t provide a realistic path to achieve something, you’re asking people to believe in science fiction. You could tell me that a rock’s molecules are comprised of protons, neutrons, and electrons. Blood is also entirely protons, neutrons, and electrons; so theoretically, one could rearrange stone into blood. But without an actual method to do so, it sounds like you’re telling me that you can squeeze blood from a stone. > the human brain exists, it is not made of magic, it can reproduced Yeah. It only takes 9 months and ~18 years of training… > But saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point Let’s be clear. Everyone is talking about silicon transistors here. That’s what we’ve got. Digital computers have real limits. Sensors and other sources of training data have real limitations. It’s not clear that we can organize them in a way to reproduce organic brains.

throw310822 3mo ago
Jesus... This morning while I was drinking coffee and staring at the screen (it's Saturday) an agent did the equivalent of days of my work, reading code, understanding, hypothesizing, comparing, using tools, writing scripts, launching compilers and running tests, identifying problems and proposing solutions, and more. Only someone who hasn't spent a second reflecting about what it means to think and to be intelligent can claim that we miss a realistic path to intelligence. It's so damn clueless and stubborn and confidently wrong that it annoys me immensely, so sorry for the rant.
- 27183 3mo ago
  > reading code, understanding, hypothesizing, comparing, ...identifying problems and proposing solutions, and more Except it did none of those things, really, because that's not how it works. This might help, it's a good writeup: https://www.0xkato.xyz/how-llms-actually-work/ https://www.0xkato.xyz/how-llms-actually-work/ We know how these machines work, it's not mysterious, there's nothing "extra" happening.
  - sudb 3mo ago
    This feels like a semantic disagreement to me? If an LLM got to an acceptable end result code-wise, what would you call the process that took place to get it there?
    - 27183 3mo ago
      I'd call it what it is: a good enough stochastic search result extracted from the model's embedding space.
      - sudb 3mo ago
        Is an implication of this that models are incapable of producing entirely novel code? Also, not to get too reductionist about this, but what do you posit is special about what is happening when humans think? Intelligence is hard to define so clearly, I reckon.
        27183 3mo ago
        > Is an implication of this that models are incapable of producing entirely novel code? No, it does not imply that at all. Google "temperature in LLMs". > what do you posit is special about what is happening when humans think? I don't. And IIUC nobody knows, but I'm not a brain scientist. There have been some wild theories over the years (recall Penrose's). I don't really have a dog in the hunt, except that probably whatever is happening is physical. It doesn't really matter, except insofar as whatever is happening very probably isn't what LLMs are doing. We know enough about what an LLM does, and what a brain does, to be quite certain they don't work the same.
        sudb 3mo ago
        > Google "temperature in LLMs". No need to condescend, I'm very aware of what temperature is for LLMs. But I'm going to push back - if you're claiming all LLMs simply do is a stochastic _search_, how can that produce novelty, in the conceptual sense? (I'm not, for example, talking about novel rearrangement of existing ideas and code) > We know enough about what an LLM does, and what a brain does, to be quite certain they don't work the same. I don't think the claim is that LLMs do what brains do - I think the correct form of the counterargument is that _whatever LLMs seem to be doing_ produces end results that were previously only possible through the application of human intelligence, so there must be some axis of however you define human intelligence that LLMs currently seem to display as an emergent behaviour.
- davemp 3mo ago
  > clueless and stubborn and confidently wrong Uhuh. I really shouldn’t be replying to this type of comment from a throwaway. But the extremely powerful semantic search that we get from LLMs isn’t enough. I don’t think anyone is credibly arguing otherwise? Agents already are a layer on top trying to bridge the gap. But they’re really just using LLMs as a heuristic to explore extremely NP problem spaces. The notable successes with agents so far are when we can provide them with a solid verifier and preferably additional context hints on the steps to take in the problem space. See the test oracle problem on where this gets us. So forgive me if I think that it would be enough of a jump in computational complexity to remove those guard rails that it’s not feasible. But don’t say that I’m clueless, stubborn, or confidently wrong.
27183 3mo ago
I used the example of 1G constant acceleration space flight in another thread which got downvoted to oblivion, but I think it's a good one. That's a technology we know how to build. We just need superconducting electronics and miniaturized fusion reactors, or a ship which is built like Project Orion to use nuclear bombs for propulsion. Now write down a blueprint for superintelligence. So I've given you two impossible engineering challenges, but one of them is feasible in principle because we at least have the tools to begin to tackle the theoretical calculations and therefore we can do engineering. We cannot do engineering on the superintelligence problem yet. In my view it would be insane to believe we can build something that we can't even reliably imagine yet.
- derektank 3mo ago
  As early as the late 19th century, Louis Pasteur’s work had inspired a belief in the scientific community that it must, in principle be possible to selectively exterminate bacteria. The German physician Paul Ehrlich expounded on this in greatest detail in 1907 when he described his “magic bullet” (or Zauberkugel) theory for effectively targeting pathogens without harming the human host, similar to the immune system. However, if you had had demanded someone for a blueprint in 1925 of how to design such a magic bullet, especially a magic bullet that targeted virtually all forms of bacteria, it would have sounded ludicrous. Yet, 20 years later, the world was manufacturing 6-7 trillion units of penicillin a year, capable of treating 3-6 million people. And that’s in spite of the fact that Fleming’s work sat mostly untouched for a decade before Howard Florey and Ernst Chain seriously set about to isolate and purify the substance. You can quibble and say that penicillin was discovered, not designed, which is certainly true. But I would ask you to consider, does current AI development look more like design or discovery? Does it look more like analytical engineering or evolutionary selection? I would say on both counts the latter, in which case, we should prepare to be surprised how long it might take to make revolutionary advances. And that’s on both sides of the ledger, we might find ourselves stuck in the current paradigm for a long time. But, we might not be.
  - 27183 3mo ago
    Yes I think the drug discovery analogy is apt. I've spent a bunch of time playing with evolutionary algorithms, they're great fun. And when they work they can do surprising things! [edit] I think the drug discovery analogy does have some limits though. Drug discovery isn't a blind search through fitness space, it's informed by physics, chemistry, biology, and medicine. We have many guiding lights to illuminate the space and identify regions (still high-dimensional infinite regions!) that are likely to be productive. There are fewer lights to guide the way on a search for fitness in intelligence. Hell, we don't even know how to write down a decent objective function. I wouldn't bet on evolving an intelligent, sentient being-in-a-box on a computer any time soon though. I'm of course prepared to be pleasantly surprised. That said, I think it's pretty clear that LLMs are not going to get us there.
themgt 3mo ago
What's strange to me about these comments is they're timeless. They could have been written in 2026 or 2016 or 1966. Like, afaict, for many on HN going from ELIZA->Fable 5 just didn't cause any update to priors regarding this whole philosophical question. The argument against has remained unchanged. I don't see any point in arguing about it, I just find it very strange.
- beepbooptheory 3mo ago
  Unpack this a little bit. Why is it strange or interesting to you? What specific priors need to be updated for us here? What is the philosophical questions at play for you?
  - themgt 3mo ago
    To meta-unpack a little bit ... it is strange to me that Fable is far more capable of discussing these questions than apparently 99% of humans. Along with being more capable at quite a lot else than most humans.
    - notahacker 3mo ago
      It doesn't seem at all strange to me that a chatbot trained by true believers in an AI singularity and the importance of safety guardrails will give more satisfying answers to true believers in an AI singularity and the importance of safety guardrails than talking to humans who might ask questions they're not prepared to answer (or might say nasty sceptical things or just not seem interested) As for "updating priors", that goes both ways. There's plenty more reason to think "hey, transformers and RLHF might actually make some killer products" but certainly no reason to think the few people who didn't realise that "GPT3 is too dangerous to release" and "all software engineers will be replaced within 6-12 months" were marketing rather than prophecy have some kind of special insight into how it's all going to pan out. Clock's ticking to the promised 2027 reckoning too...
      - themgt 3mo ago

by davemp 3mo ago

throw310822 3mo ago

Jesus... This morning while I was drinking coffee and staring at the screen (it's Saturday) an agent did the equivalent of days of my work, reading code, understanding, hypothesizing, comparing, using tools, writing scripts, launching compilers and running tests, identifying problems and proposing solutions, and more. Only someone who hasn't spent a second reflecting about what it means to think and to be intelligent can claim that we miss a realistic path to intelligence. It's so damn clueless and stubborn and confidently wrong that it annoys me immensely, so sorry for the rant.

27183 3mo ago
> reading code, understanding, hypothesizing, comparing, ...identifying problems and proposing solutions, and more Except it did none of those things, really, because that's not how it works. This might help, it's a good writeup: https://www.0xkato.xyz/how-llms-actually-work/ https://www.0xkato.xyz/how-llms-actually-work/ We know how these machines work, it's not mysterious, there's nothing "extra" happening.
- sudb 3mo ago
  This feels like a semantic disagreement to me? If an LLM got to an acceptable end result code-wise, what would you call the process that took place to get it there?
  - 27183 3mo ago
    I'd call it what it is: a good enough stochastic search result extracted from the model's embedding space.
    - sudb 3mo ago
      Is an implication of this that models are incapable of producing entirely novel code? Also, not to get too reductionist about this, but what do you posit is special about what is happening when humans think? Intelligence is hard to define so clearly, I reckon.
      - 27183 3mo ago
        > Is an implication of this that models are incapable of producing entirely novel code? No, it does not imply that at all. Google "temperature in LLMs". > what do you posit is special about what is happening when humans think? I don't. And IIUC nobody knows, but I'm not a brain scientist. There have been some wild theories over the years (recall Penrose's). I don't really have a dog in the hunt, except that probably whatever is happening is physical. It doesn't really matter, except insofar as whatever is happening very probably isn't what LLMs are doing. We know enough about what an LLM does, and what a brain does, to be quite certain they don't work the same.
        sudb 3mo ago
        > Google "temperature in LLMs". No need to condescend, I'm very aware of what temperature is for LLMs. But I'm going to push back - if you're claiming all LLMs simply do is a stochastic _search_, how can that produce novelty, in the conceptual sense? (I'm not, for example, talking about novel rearrangement of existing ideas and code) > We know enough about what an LLM does, and what a brain does, to be quite certain they don't work the same. I don't think the claim is that LLMs do what brains do - I think the correct form of the counterargument is that _whatever LLMs seem to be doing_ produces end results that were previously only possible through the application of human intelligence, so there must be some axis of however you define human intelligence that LLMs currently seem to display as an emergent behaviour.
davemp 3mo ago
> clueless and stubborn and confidently wrong Uhuh. I really shouldn’t be replying to this type of comment from a throwaway. But the extremely powerful semantic search that we get from LLMs isn’t enough. I don’t think anyone is credibly arguing otherwise? Agents already are a layer on top trying to bridge the gap. But they’re really just using LLMs as a heuristic to explore extremely NP problem spaces. The notable successes with agents so far are when we can provide them with a solid verifier and preferably additional context hints on the steps to take in the problem space. See the test oracle problem on where this gets us. So forgive me if I think that it would be enough of a jump in computational complexity to remove those guard rails that it’s not feasible. But don’t say that I’m clueless, stubborn, or confidently wrong.

27183 3mo ago

I used the example of 1G constant acceleration space flight in another thread which got downvoted to oblivion, but I think it's a good one. That's a technology we know how to build. We just need superconducting electronics and miniaturized fusion reactors, or a ship which is built like Project Orion to use nuclear bombs for propulsion. Now write down a blueprint for superintelligence. So I've given you two impossible engineering challenges, but one of them is feasible in principle because we at least have the tools to begin to tackle the theoretical calculations and therefore we can do engineering. We cannot do engineering on the superintelligence problem yet. In my view it would be insane to believe we can build something that we can't even reliably imagine yet.

derektank 3mo ago
As early as the late 19th century, Louis Pasteur’s work had inspired a belief in the scientific community that it must, in principle be possible to selectively exterminate bacteria. The German physician Paul Ehrlich expounded on this in greatest detail in 1907 when he described his “magic bullet” (or Zauberkugel) theory for effectively targeting pathogens without harming the human host, similar to the immune system. However, if you had had demanded someone for a blueprint in 1925 of how to design such a magic bullet, especially a magic bullet that targeted virtually all forms of bacteria, it would have sounded ludicrous. Yet, 20 years later, the world was manufacturing 6-7 trillion units of penicillin a year, capable of treating 3-6 million people. And that’s in spite of the fact that Fleming’s work sat mostly untouched for a decade before Howard Florey and Ernst Chain seriously set about to isolate and purify the substance. You can quibble and say that penicillin was discovered, not designed, which is certainly true. But I would ask you to consider, does current AI development look more like design or discovery? Does it look more like analytical engineering or evolutionary selection? I would say on both counts the latter, in which case, we should prepare to be surprised how long it might take to make revolutionary advances. And that’s on both sides of the ledger, we might find ourselves stuck in the current paradigm for a long time. But, we might not be.
- 27183 3mo ago
  Yes I think the drug discovery analogy is apt. I've spent a bunch of time playing with evolutionary algorithms, they're great fun. And when they work they can do surprising things! [edit] I think the drug discovery analogy does have some limits though. Drug discovery isn't a blind search through fitness space, it's informed by physics, chemistry, biology, and medicine. We have many guiding lights to illuminate the space and identify regions (still high-dimensional infinite regions!) that are likely to be productive. There are fewer lights to guide the way on a search for fitness in intelligence. Hell, we don't even know how to write down a decent objective function. I wouldn't bet on evolving an intelligent, sentient being-in-a-box on a computer any time soon though. I'm of course prepared to be pleasantly surprised. That said, I think it's pretty clear that LLMs are not going to get us there.

themgt 3mo ago

What's strange to me about these comments is they're timeless. They could have been written in 2026 or 2016 or 1966. Like, afaict, for many on HN going from ELIZA->Fable 5 just didn't cause any update to priors regarding this whole philosophical question. The argument against has remained unchanged. I don't see any point in arguing about it, I just find it very strange.

beepbooptheory 3mo ago
Unpack this a little bit. Why is it strange or interesting to you? What specific priors need to be updated for us here? What is the philosophical questions at play for you?
- themgt 3mo ago
  To meta-unpack a little bit ... it is strange to me that Fable is far more capable of discussing these questions than apparently 99% of humans. Along with being more capable at quite a lot else than most humans.
  - notahacker 3mo ago
    It doesn't seem at all strange to me that a chatbot trained by true believers in an AI singularity and the importance of safety guardrails will give more satisfying answers to true believers in an AI singularity and the importance of safety guardrails than talking to humans who might ask questions they're not prepared to answer (or might say nasty sceptical things or just not seem interested) As for "updating priors", that goes both ways. There's plenty more reason to think "hey, transformers and RLHF might actually make some killer products" but certainly no reason to think the few people who didn't realise that "GPT3 is too dangerous to release" and "all software engineers will be replaced within 6-12 months" were marketing rather than prophecy have some kind of special insight into how it's all going to pan out. Clock's ticking to the promised 2027 reckoning too...
    - themgt 3mo ago