3 ms·

It's nuts how well "Superintelligence: The Idea That Eats Smart People"[0] aged. That talk is a decade old by now and still hits just as hard as it did back the

It's nuts how well "Superintelligence: The Idea That Eats Smart People"[0] aged. That talk is a decade old by now and still hits just as hard as it did back then, despite the incredible advances made in AI in the meantime. [0] https://idlewords.com/talks/superintelligence.htm https://idlewords.com/talks/superintelligence.htm

taurath 3mo ago
Yeah I think its possible that for many folks its the first time they're coming up on these concepts, and it troubles them in the same way that the concept of death troubles them (and me, to be clear!). For me its as simple as watching how people talk, and seeing how in every single case whatever the next thing is, if you believe It, there is only ever justification of doubling down, doing more, going deeper, reducing any doubt. These are not scientists, they're business people and salespeople, and a few optimists having recently on paper solved all their worldly financial needs. Even if one throws that aside, spending time exploring and building with the most state of the art LLMs is just as instructive. I'm watching the implementation - whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. Sit in front and observe ideas turn to the samey intellectual, high-syllable mush. Its productive, but not in any way that's promised.
- TheOtherHobbes 3mo ago
  Did you actually read the text? OPs are calling that Plan D. They're proposing an alternative, which is a global brake on frontier AI research to keep the basilisk in its jar until we work out what we're dealing with and how to handle it.
  - layla5alive 3mo ago
    No, they're proposing a spying panopticon and state control of global resource distribution - specifically general purpose compute - including seizing and destroying GPUs. They're proposing a totalitarian global dictatorship controlling computing hardware and software. Lest you think I'm being hyperbolic: https://ai-2040.com/supplements/covert-ai-projects https://ai-2040.com/supplements/covert-ai-projects This is arsonists selling fire insurance.
    - esafak 3mo ago
      A nasty global inspections regime just like we have for nuclear weapons, which are less dangerous than this. Oh no ...
      - layla5alive 3mo ago
        Nuclear weapons are less dangerous than compute? Are you listening to yourself? Humans employing this kind of thinking and desire to control other people is the genuine danger here.
- YeGoblynQueenne 3mo ago
  >> Even if one throws that aside, spending time exploring and building with the most state of the art LLMs is just as instructive. I'm watching the implementation - whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. Sit in front and observe ideas turn to the samey intellectual, high-syllable mush. Its productive, but not in any way that's promised. Important point. LLMs were early on hailed as the first general-puprose AIs that can perform any task (remember "Sparks of AGI"?). Today they're increasingly promoted for specialised applications - coding, as a for instance.
  - Leynos 3mo ago
    While there are some coding focused models (composer, for example), the majority of frontier models are pitched as general purpose. The coding harnesses for Claude and GPT are even being repurposed as general purpose knowledge work harnesses.
    - YeGoblynQueenne 3mo ago
      No, you're right of course, but I have a feeling it's much easier to sell a system with a clear goal, like "this LLM generates code" or "this LLM solves math problems". Even if the underlying model is a general purpose one. I think there's always a question, when one has a product, of "what does it do?". "This thing does everything you want it to" is not a great way to sell something. More to the point, even models marketed as general purpose are clearly trained on specific tasks. That is, the AI companies want to promote their systems as general purpose but they also want to make them good at specific tasks, because that supports their marketing story, that those are general-purpose systems that are so powerful they can even do maths and science. Or maybe code, maths, and (maybe) science are just the things they find it easy to train their models on, for different reasons and in different ways. You can also get a feel about the things they have tried to do and failed, e.g. real-world autonomy isn't really working (or not working yet, who knows), so OpenAI is not trying to sell an embodied generalist agent say, that can clean your hose, do your laundry and teach your kids maths and science on the side. I mean even generating text is a very narrow task, in the general sense, compared to all the things that humans can do (never mind other animals) it's just that we use text so much and for so many things that there's an endless list of applications for a text generator; as we have all found out.
- dmitriy_ko 3mo ago
  > whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. As usual, AI skeptics are moving goal posts. Modern LLMs are on a completely different level in terms of how GENERAL they are vs anything pre-LLM. You can give it a completely novel puzzle and it will solve it. 5+ years ago you had to train NN to solve particular type of puzzle.
  - deleted 3mo ago
    [deleted]
deleted 3mo ago
[deleted]
beshrkayali 3mo ago
This is hilariously true > AI risk is string theory for computer programmers. It's fun to think about, interesting, and completely inaccessible to experiment given our current technology. You can build crystal palaces of thought, working from first principles, then climb up inside them and pull the ladder up behind you. People who can reach preposterous conclusions from a long chain of abstract reasoning, and feel confident in their truth, are the wrong people to be running a culture. I understand how people running in the same scene fall into the echo chamber effect and get gulped into the cult, but why does everybody want to be a prophet?
- walrus01 3mo ago
  Being a prophet is probably great until you suddenly find yourself building a fortified compound in Waco, Texas and purchasing black market full auto machine guns.
  - saltwatercowboy 3mo ago
    Thank you for contacting Gunmetal Ranch, a legitimate 501(c). If your call is related to the class-action "most dangerous game" settlement, please hold for a cowpoke.
- int3trap 3mo ago
  > but why does everybody want to be a prophet? Its not everyone building crystal palaces in their mind, they're all building fortresses. And they can't be wrong in their fortress or it breaks their world view which they cannot accept.
- 0xDEAFBEAD 3mo ago
  Theorizing about nuclear winter is somewhat similar, in the sense of being inaccessible to experiment. Does that mean we should disregard the possibility of nuclear winter?
  - m4rtink 3mo ago
    As with using the Orion drive for launch and landing on Erth is possible, test nuclear winter is certainly possible. But as with the former, you won't be very popular among the survivors (if any).
GCUMstlyHarmls 3mo ago
Video link, as the link on the page is dead. https://www.youtube.com/watch?v=kErHiET5YPw https://www.youtube.com/watch?v=kErHiET5YPw
casey2 3mo ago
and upon rereading completely holds up technically we are still passing in massive data into simple networks giving no opportunity for introspection or recursive self improvement.
0xDEAFBEAD 3mo ago
>despite the incredible advances made in AI in the meantime So the goalposts will be moved whenever necessary in that case? No amount of incredible advances in AI will ever get skeptical HN commenters to take AI's implications seriously? "The Gish gallops will continue until the nagging doubts have been silenced"
tim333 3mo ago
I always thought it a bit pessimistic on the upsides like robot to build us palaces, immortality, conquer the galaxy type stuff along the lines of the "Grandiosity" bit in the essay. It's a shame we have amazing tech possibilities and most people are either saying either it's rubbish and won't work or it'll bring doom. Aside from being boring they are probably factually wrong. Bit like saying early medical knowledge was rubbish or will doom us by summoning demons while really life expectancy at birth went from twenty something in Roman times to eighty odd now.

taurath 3mo ago

Yeah I think its possible that for many folks its the first time they're coming up on these concepts, and it troubles them in the same way that the concept of death troubles them (and me, to be clear!). For me its as simple as watching how people talk, and seeing how in every single case whatever the next thing is, if you believe It, there is only ever justification of doubling down, doing more, going deeper, reducing any doubt. These are not scientists, they're business people and salespeople, and a few optimists having recently on paper solved all their worldly financial needs. Even if one throws that aside, spending time exploring and building with the most state of the art LLMs is just as instructive. I'm watching the implementation - whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. Sit in front and observe ideas turn to the samey intellectual, high-syllable mush. Its productive, but not in any way that's promised.

TheOtherHobbes 3mo ago
Did you actually read the text? OPs are calling that Plan D. They're proposing an alternative, which is a global brake on frontier AI research to keep the basilisk in its jar until we work out what we're dealing with and how to handle it.
- layla5alive 3mo ago
  No, they're proposing a spying panopticon and state control of global resource distribution - specifically general purpose compute - including seizing and destroying GPUs. They're proposing a totalitarian global dictatorship controlling computing hardware and software. Lest you think I'm being hyperbolic: https://ai-2040.com/supplements/covert-ai-projects https://ai-2040.com/supplements/covert-ai-projects This is arsonists selling fire insurance.
  - esafak 3mo ago
    A nasty global inspections regime just like we have for nuclear weapons, which are less dangerous than this. Oh no ...
    - layla5alive 3mo ago
      Nuclear weapons are less dangerous than compute? Are you listening to yourself? Humans employing this kind of thinking and desire to control other people is the genuine danger here.
YeGoblynQueenne 3mo ago
>> Even if one throws that aside, spending time exploring and building with the most state of the art LLMs is just as instructive. I'm watching the implementation - whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. Sit in front and observe ideas turn to the samey intellectual, high-syllable mush. Its productive, but not in any way that's promised. Important point. LLMs were early on hailed as the first general-puprose AIs that can perform any task (remember "Sparks of AGI"?). Today they're increasingly promoted for specialised applications - coding, as a for instance.
- Leynos 3mo ago
  While there are some coding focused models (composer, for example), the majority of frontier models are pitched as general purpose. The coding harnesses for Claude and GPT are even being repurposed as general purpose knowledge work harnesses.
  - YeGoblynQueenne 3mo ago
    No, you're right of course, but I have a feeling it's much easier to sell a system with a clear goal, like "this LLM generates code" or "this LLM solves math problems". Even if the underlying model is a general purpose one. I think there's always a question, when one has a product, of "what does it do?". "This thing does everything you want it to" is not a great way to sell something. More to the point, even models marketed as general purpose are clearly trained on specific tasks. That is, the AI companies want to promote their systems as general purpose but they also want to make them good at specific tasks, because that supports their marketing story, that those are general-purpose systems that are so powerful they can even do maths and science. Or maybe code, maths, and (maybe) science are just the things they find it easy to train their models on, for different reasons and in different ways. You can also get a feel about the things they have tried to do and failed, e.g. real-world autonomy isn't really working (or not working yet, who knows), so OpenAI is not trying to sell an embodied generalist agent say, that can clean your hose, do your laundry and teach your kids maths and science on the side. I mean even generating text is a very narrow task, in the general sense, compared to all the things that humans can do (never mind other animals) it's just that we use text so much and for so many things that there's an endless list of applications for a text generator; as we have all found out.
dmitriy_ko 3mo ago
> whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. As usual, AI skeptics are moving goal posts. Modern LLMs are on a completely different level in terms of how GENERAL they are vs anything pre-LLM. You can give it a completely novel puzzle and it will solve it. 5+ years ago you had to train NN to solve particular type of puzzle.
- deleted 3mo ago
  [deleted]

deleted 3mo ago

[deleted]

beshrkayali 3mo ago

This is hilariously true > AI risk is string theory for computer programmers. It's fun to think about, interesting, and completely inaccessible to experiment given our current technology. You can build crystal palaces of thought, working from first principles, then climb up inside them and pull the ladder up behind you. People who can reach preposterous conclusions from a long chain of abstract reasoning, and feel confident in their truth, are the wrong people to be running a culture. I understand how people running in the same scene fall into the echo chamber effect and get gulped into the cult, but why does everybody want to be a prophet?

walrus01 3mo ago
Being a prophet is probably great until you suddenly find yourself building a fortified compound in Waco, Texas and purchasing black market full auto machine guns.
- saltwatercowboy 3mo ago
  Thank you for contacting Gunmetal Ranch, a legitimate 501(c). If your call is related to the class-action "most dangerous game" settlement, please hold for a cowpoke.
int3trap 3mo ago
> but why does everybody want to be a prophet? Its not everyone building crystal palaces in their mind, they're all building fortresses. And they can't be wrong in their fortress or it breaks their world view which they cannot accept.
0xDEAFBEAD 3mo ago
Theorizing about nuclear winter is somewhat similar, in the sense of being inaccessible to experiment. Does that mean we should disregard the possibility of nuclear winter?
- m4rtink 3mo ago
  As with using the Orion drive for launch and landing on Erth is possible, test nuclear winter is certainly possible. But as with the former, you won't be very popular among the survivors (if any).

GCUMstlyHarmls 3mo ago

Video link, as the link on the page is dead. https://www.youtube.com/watch?v=kErHiET5YPw https://www.youtube.com/watch?v=kErHiET5YPw

casey2 3mo ago

and upon rereading completely holds up technically we are still passing in massive data into simple networks giving no opportunity for introspection or recursive self improvement.

0xDEAFBEAD 3mo ago

>despite the incredible advances made in AI in the meantime So the goalposts will be moved whenever necessary in that case? No amount of incredible advances in AI will ever get skeptical HN commenters to take AI's implications seriously? "The Gish gallops will continue until the nagging doubts have been silenced"

tim333 3mo ago

I always thought it a bit pessimistic on the upsides like robot to build us palaces, immortality, conquer the galaxy type stuff along the lines of the "Grandiosity" bit in the essay. It's a shame we have amazing tech possibilities and most people are either saying either it's rubbish and won't work or it'll bring doom. Aside from being boring they are probably factually wrong. Bit like saying early medical knowledge was rubbish or will doom us by summoning demons while really life expectancy at birth went from twenty something in Roman times to eighty odd now.