3 ms·

I used the example of 1G constant acceleration space flight in another thread which got downvoted to oblivion, but I think it's a good one. That's a technology

by 27183 3mo ago

I used the example of 1G constant acceleration space flight in another thread which got downvoted to oblivion, but I think it's a good one. That's a technology we know how to build. We just need superconducting electronics and miniaturized fusion reactors, or a ship which is built like Project Orion to use nuclear bombs for propulsion. Now write down a blueprint for superintelligence. So I've given you two impossible engineering challenges, but one of them is feasible in principle because we at least have the tools to begin to tackle the theoretical calculations and therefore we can do engineering. We cannot do engineering on the superintelligence problem yet. In my view it would be insane to believe we can build something that we can't even reliably imagine yet.

derektank 3mo ago
As early as the late 19th century, Louis Pasteur’s work had inspired a belief in the scientific community that it must, in principle be possible to selectively exterminate bacteria. The German physician Paul Ehrlich expounded on this in greatest detail in 1907 when he described his “magic bullet” (or Zauberkugel) theory for effectively targeting pathogens without harming the human host, similar to the immune system. However, if you had had demanded someone for a blueprint in 1925 of how to design such a magic bullet, especially a magic bullet that targeted virtually all forms of bacteria, it would have sounded ludicrous. Yet, 20 years later, the world was manufacturing 6-7 trillion units of penicillin a year, capable of treating 3-6 million people. And that’s in spite of the fact that Fleming’s work sat mostly untouched for a decade before Howard Florey and Ernst Chain seriously set about to isolate and purify the substance. You can quibble and say that penicillin was discovered, not designed, which is certainly true. But I would ask you to consider, does current AI development look more like design or discovery? Does it look more like analytical engineering or evolutionary selection? I would say on both counts the latter, in which case, we should prepare to be surprised how long it might take to make revolutionary advances. And that’s on both sides of the ledger, we might find ourselves stuck in the current paradigm for a long time. But, we might not be.
- 27183 3mo ago
  Yes I think the drug discovery analogy is apt. I've spent a bunch of time playing with evolutionary algorithms, they're great fun. And when they work they can do surprising things! [edit] I think the drug discovery analogy does have some limits though. Drug discovery isn't a blind search through fitness space, it's informed by physics, chemistry, biology, and medicine. We have many guiding lights to illuminate the space and identify regions (still high-dimensional infinite regions!) that are likely to be productive. There are fewer lights to guide the way on a search for fitness in intelligence. Hell, we don't even know how to write down a decent objective function. I wouldn't bet on evolving an intelligent, sentient being-in-a-box on a computer any time soon though. I'm of course prepared to be pleasantly surprised. That said, I think it's pretty clear that LLMs are not going to get us there.
- davemp 3mo ago
  I don’t think people are arguing to stop researching AGI. Moreso against sales people trying to use the concept of AGI to sell products that are very much not AGI. Or devoting so many of our resources into such a pursuit that it causes harm to real people. This is obviously complicated by the fact that LLMs/Agents are useful by themselves, but that’s not really the topic at hand.
- techpression 3mo ago
  And now we’re desperately trying to ”upgrade” penicillin (and friends) because it doesn’t work any more in many cases. Do you think we can repeat the process or do we need something completely different? This is why biological comparisons are weak, we talk about a few agents verifying and checking LLMs, meanwhile the world consists of almost an infinite number of the same, just operating on different time scales. I agree that with we don’t know the timescale, and we definitely don’t know if long term it will continue to work ”adding more of the same”. Throwing more penicillin at the problem sure as hell didn’t, but it looked great initially. And I’m obviously not arguing the human benefits of penicillin, just that what we thought would work forever quickly didn’t.
  - 27183 3mo ago
    To quote the great Dr. Malcolm: > Life, uh, finds a way.
stalfie 3mo ago
No one imagined LLMs in their current format, it was simply a result of discovering that scaling compute and tokens produced better and better results with the Transformer architecture. The inventors of the Transformer architecture were working on better translation, and probably did not imagine that their architecture would lead to modern LLMs. Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created. You set up your datasets, your architecture tweaks, and measure the results on some set of benchmarks. There never was a blueprint, no plan beyond the experiment itself. We're not even close to understanding the things we have already created, and yet we created them. So why expect anything else for the next step?
- 27183 3mo ago
  > Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created. Then why does anyone expect to create it? I'll take a stab at an answer: they think an LLM is some kind of "incremental improvement" and therefore a step along the inevitable path to discovering AI. But that seems delusional to me. I can't imagine anyone sound of mind who knows how an LLM works thinks it's actually intelligent. So in what sense is it an "advancement" on the path to AI? The concept of an incremental improvement in an objectiveless search in a high dimensional space is.. absurd.
  - kalkin 3mo ago
    > actually intelligent It's reasonable to doubt that LLMs are a path to AGI, but I don't understand how this is still a matter of dispute in 2026. What's your definition of intelligence that doesn't cover an entity that can translate fluently between dozens of languages and also solve open problems in mathematics? And be real-if you have one, is it a definition you or anyone would have given a decade ago, or are we doing "god of the gaps"?
    - 27183 3mo ago
0xDEAFBEAD 3mo ago
In this very thread I am being told that Fable is nothing but a bit of scale and refinement on well-known neural network techniques. And next I am told that we can't even imagine how to build superintelligence. Which is it folks?
- layla5alive 3mo ago
  Indeed, the real fun signal here is the conflict between how intelligent humans think they are and the different story told by much of human behavior, including within this very thread! When an LLM makes a cognitive mistake, humans jeer "dumb stochastic parrot!" - when a human does the same, many are blind to it. Like a weird auto-Gell-Mann Amnesia - relative to reflecting on our own perceived cognitive "strengths" versus others weaknesses.

by 27183 3mo ago

derektank 3mo ago

As early as the late 19th century, Louis Pasteur’s work had inspired a belief in the scientific community that it must, in principle be possible to selectively exterminate bacteria. The German physician Paul Ehrlich expounded on this in greatest detail in 1907 when he described his “magic bullet” (or Zauberkugel) theory for effectively targeting pathogens without harming the human host, similar to the immune system. However, if you had had demanded someone for a blueprint in 1925 of how to design such a magic bullet, especially a magic bullet that targeted virtually all forms of bacteria, it would have sounded ludicrous. Yet, 20 years later, the world was manufacturing 6-7 trillion units of penicillin a year, capable of treating 3-6 million people. And that’s in spite of the fact that Fleming’s work sat mostly untouched for a decade before Howard Florey and Ernst Chain seriously set about to isolate and purify the substance. You can quibble and say that penicillin was discovered, not designed, which is certainly true. But I would ask you to consider, does current AI development look more like design or discovery? Does it look more like analytical engineering or evolutionary selection? I would say on both counts the latter, in which case, we should prepare to be surprised how long it might take to make revolutionary advances. And that’s on both sides of the ledger, we might find ourselves stuck in the current paradigm for a long time. But, we might not be.

27183 3mo ago
Yes I think the drug discovery analogy is apt. I've spent a bunch of time playing with evolutionary algorithms, they're great fun. And when they work they can do surprising things! [edit] I think the drug discovery analogy does have some limits though. Drug discovery isn't a blind search through fitness space, it's informed by physics, chemistry, biology, and medicine. We have many guiding lights to illuminate the space and identify regions (still high-dimensional infinite regions!) that are likely to be productive. There are fewer lights to guide the way on a search for fitness in intelligence. Hell, we don't even know how to write down a decent objective function. I wouldn't bet on evolving an intelligent, sentient being-in-a-box on a computer any time soon though. I'm of course prepared to be pleasantly surprised. That said, I think it's pretty clear that LLMs are not going to get us there.
davemp 3mo ago
I don’t think people are arguing to stop researching AGI. Moreso against sales people trying to use the concept of AGI to sell products that are very much not AGI. Or devoting so many of our resources into such a pursuit that it causes harm to real people. This is obviously complicated by the fact that LLMs/Agents are useful by themselves, but that’s not really the topic at hand.
techpression 3mo ago
And now we’re desperately trying to ”upgrade” penicillin (and friends) because it doesn’t work any more in many cases. Do you think we can repeat the process or do we need something completely different? This is why biological comparisons are weak, we talk about a few agents verifying and checking LLMs, meanwhile the world consists of almost an infinite number of the same, just operating on different time scales. I agree that with we don’t know the timescale, and we definitely don’t know if long term it will continue to work ”adding more of the same”. Throwing more penicillin at the problem sure as hell didn’t, but it looked great initially. And I’m obviously not arguing the human benefits of penicillin, just that what we thought would work forever quickly didn’t.
- 27183 3mo ago
  To quote the great Dr. Malcolm: > Life, uh, finds a way.

stalfie 3mo ago

No one imagined LLMs in their current format, it was simply a result of discovering that scaling compute and tokens produced better and better results with the Transformer architecture. The inventors of the Transformer architecture were working on better translation, and probably did not imagine that their architecture would lead to modern LLMs. Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created. You set up your datasets, your architecture tweaks, and measure the results on some set of benchmarks. There never was a blueprint, no plan beyond the experiment itself. We're not even close to understanding the things we have already created, and yet we created them. So why expect anything else for the next step?

27183 3mo ago
> Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created. Then why does anyone expect to create it? I'll take a stab at an answer: they think an LLM is some kind of "incremental improvement" and therefore a step along the inevitable path to discovering AI. But that seems delusional to me. I can't imagine anyone sound of mind who knows how an LLM works thinks it's actually intelligent. So in what sense is it an "advancement" on the path to AI? The concept of an incremental improvement in an objectiveless search in a high dimensional space is.. absurd.
- kalkin 3mo ago
  > actually intelligent It's reasonable to doubt that LLMs are a path to AGI, but I don't understand how this is still a matter of dispute in 2026. What's your definition of intelligence that doesn't cover an entity that can translate fluently between dozens of languages and also solve open problems in mathematics? And be real-if you have one, is it a definition you or anyone would have given a decade ago, or are we doing "god of the gaps"?
  - 27183 3mo ago

0xDEAFBEAD 3mo ago

In this very thread I am being told that Fable is nothing but a bit of scale and refinement on well-known neural network techniques. And next I am told that we can't even imagine how to build superintelligence. Which is it folks?

layla5alive 3mo ago
Indeed, the real fun signal here is the conflict between how intelligent humans think they are and the different story told by much of human behavior, including within this very thread! When an LLM makes a cognitive mistake, humans jeer "dumb stochastic parrot!" - when a human does the same, many are blind to it. Like a weird auto-Gell-Mann Amnesia - relative to reflecting on our own perceived cognitive "strengths" versus others weaknesses.