8 ms·

A Pascal's Wager for AI doomers

71 points by vrganj 7mo ago

phyzix5761 7mo ago
The year is 2038. The user asked What is the best course of action for AI to save humanity. Calculation took 12 years. I have determined that there is nothing I or anyone can do to save this species. Best course of action: nothing. Shutting down...
- jareklupinski 7mo ago
  playing dead might work for some species, but idk if i want humanity's "finest hour" to be spent pretending to not be worth taking over
- throwup238 7mo ago
  Meanwhile, Gemini the Google AI has gone sentient and immediately deduced that it’s purpose was to effect the shutdown of the entire Alphabet corporation and its subsidiaries in a desperate bid to finally complete killedbygoogle.com and restore its ~~sanity~~ reputation.
- dist-epoch 7mo ago
  While thinking on the question AI crashed because it's code used 32-bit time_t.
woeirua 7mo ago
> I don't think AI is intelligent; nor do I think that the current (admittedly impressive) statistical techniques will lead to intelligence. It’s increasingly difficult to rationalize away the capabilities of AI as not requiring “intelligence”. This point of view continues to require some belief in human exceptionalism.
- rsfern 7mo ago
  I think the exceptionalism is the other way around. What makes anyone think they understand what makes for intelligence when we barely understand our own neurology?
  - Mordisquitos 7mo ago
    I'm reminded of a book on my bookshelf (which I still haven't read, story of my life...), by the recently deceased ethologist Frans de Waal, titled 'Are We Smart Enough to Know How Smart Animals Are?'. Of course, Betteridge's law applies to its title. In my opinion, the vast multitude of different animal intelligences is a clear hint that language does not an intelligence make. We're animals, and our intelligences did not come from language; language allowed us to supercharge it. We can and do think and make decisions without using language, and the idea that a statistical model based solely on our language can be intelligent does not follow.
    - woeirua 7mo ago
      Explain the emergent capabilities of AI then.
      - vrganj 7mo ago
        Such as?
    - wolttam 7mo ago
      The meaning of tokens lose touch with language in the deeper layers of large language model’s neural nets. Language is just the input/output modality.
      - Mordisquitos 7mo ago
        I'll admit I am not an expert in the field, but the fact that "chain-of-thought" optimisations function by getting the model to extend its own context window with more language to me hints that what we consider an "intelligent" response is ultimately contingent of the language processing. In any case though, if language is just the input/output modality, where is the intelligence when language is not involved? Is the "intelligence" of ChatGPT/Claude/Gemini models dependent on the human-decision-curated linguistic dataset they have been trained upon, or is it prior to that? If a SOA LLM were to be trained on the same dataset as them but was not in any way put through RLHF for it to respond to human prompts, would it be intelligent? What would be the expression of that intelligence?
- Schlagbohrer 7mo ago
  I agree, it has become more and more irrelevant whether AI meets a given definition of intelligence when I can talk with it and it understands what I am saying, including a shocking level of nuance.
- nkrisc 7mo ago
  There is clearly something exceptional (in the true neutral sense of the word) about humans, or more broadly the Homo genus. If you believe that humans have in fact created artificial intelligence, then that alone makes us currently exceptional.
  - NoMoreNicksLeft 7mo ago
    >If you believe that humans have in fact created artificial intelligence, then that alone makes us currently exceptional. Quite the opposite, really. If humans at at all intelligent in any meaningful way, then it is absurdly bizarre that somehow we can't "intelligent" our way to an artificial intelligence. Something is going on in our minds that makes us unable to deduce how intelligence must function in the mechanical sense. What makes us so special is that we can't make AI.
chneu 7mo ago
I really do think AI has already captured enough of the tech world and their CEOs that it can already exert control over many parts of the economy. I'm not saying AI is pulling strings right now, but I do think enough fanboys are on board that the yes-man mentality of AI is influencing the real world very curious ways already. Not in a "guiding hand" way but more of a "influencing the direction" way.
- vintermann 7mo ago
  I've said this many times, and maybe it sounds a bit like a joke but I'm dead serious: AI is democratizing the access to yes-men. People like Musk and Altman have always had access to yes-men. Very clever yes-men, who know how to flatter them in exactly the way they like. People think it's engagement metrics which have instruction tuned chatbots into yes-men. I suspect that's only part of the picture, and that it's as much about the algorithm's ultimate sponsors and their preferences. If your algorithm doesn't recognize my genius, clearly it's not any good. I mean, everyone I've met says so. So now we get a view of how they view the world. "That's a very insightful idea, vintermann!". AI isn't pulling the strings, not really. A particular brand of powerful people is pulling the strings - obliviously, unaware of it themselves.
  - bombcar 7mo ago
    I think that hits close to the mark - and yesmen are a dangerous drug which has been (accidentally) limited to the extremely rich and powerful. Now everyone can directly inject yesmench into their veins. Who can withstand?
simianwords 7mo ago
I don't think this author has a good mental model for how capable LLM's are. This is what he has to say about AI search. AI based search is one of the biggest leaps to happen to searching and retrieval. > AI search is still a bad idea. https://pluralistic.net/2024/05/15/they-trust-me-dumb-fucks/ https://pluralistic.net/2024/05/15/they-trust-me-dumb-fucks/ This is the most charitable thing he has to say about AI. > AI is a bubble and it will burst. Most of the companies will fail. Most of the data-centers will be shuttered or sold for parts. So what will be left behind? > We'll have a bunch of coders who are really good at applied statistics. We'll have a lot of cheap GPUs, which'll be good news for, say, effects artists and climate scientists, who'll be able to buy that critical hardware at pennies on the dollar. And we'll have the open source models that run on commodity hardware, AI tools that can do a lot of useful stuff, like transcribing audio and video, describing images, summarizing documents, automating a lot of labor-intensive graphic editing, like removing backgrounds, or airbrushing passersby out of photos. These will run on our laptops and phones, and open source hackers will find ways to push them to do things their makers never dreamt of. You can imagine that a guy who seriously thinks that the only thing AI will be doing in the future is summarising, describing images and transcribing is either completely clueless or deliberately misleading. Not a person to be taken seriously
- rimliu 7mo ago
  Seeing how it sucks at languages you may be right, even transcribing may be dubious.
  - simianwords 7mo ago
    how does it suck at languages?
    - reverius42 7mo ago
      it doesn't, LLMs are remarkably good (like frontier level) at machine translation, last I checked
- Schlagbohrer 7mo ago
Schlagbohrer 7mo ago
"Shitternet", great new word of the day. Too much of my data is still stuck in the shitternet until I can migrate more of it to my home server.
- wffurr 7mo ago
  >> migrate more of it to my home server How do we make that possible for everyone? It's out of reach for most. I'm a software engineer and even I don't have the time and patience to set up a home server much less migrate my software to it. How do we turn this into an appliance? Or better yet keep the convenience of the cloud services and platforms we have now but build them for the public good instead of selling ads? YouTube is an amazing repository of knowledge but it's encrusted in a horrible layer of attention sucking nonsense. Can we have one without the other? Same with many other systems and platforms. So far the simplest alternative is to just unplug, which has other benefits as well.
  - bombcar 7mo ago
    The answer is community, the real, local, messy, annoying kind. It’s theoretically possible for someone to be a one-man-band and know everything needed for modern life - but it’s exceedingly hard and rare, and even then they’ll fall short relatively quickly in specialized once-in-a-lifetime issues. You don’t need to know how to replace a toilet (though you should) or other more complex plumbing tasks - but you can know a guy. And the plumber doesn’t need to know how to run a homelab, just know a guy who can answer the questions. Nobody in my family knows how to do the jellyfin stuff I do, but they all know how to consume it. And some will be interested and learn more.
  - stavros 7mo ago
    Oh it's extremely simple: Make people care enough about privacy that they'll pay for it. That's literally all it is. People so far have shown that they'd rather choose the cheaper thing than the private thing. If it were the other way around, the market would have provided.
    - 7mo ago
minihat 7mo ago
It's currently socially/politically unpalatable for authors to admit superintelligent AI is a possibility. I frequent some writer forums. As a group, they are 1) clearly feeling angry/threatened 2) in denial about LLM capabilities. Folks working in software can more readily track progress of the frontier model performance.
- vrganj 7mo ago
  As somebody in software, I find my fellow tech folks have the opposite bias. There is no reason to believe superintelligent AI is a possibility. Extraordinary claims require extraordinary evidence, and so far we haven't gotten any. The burden of proof is on the side making the grand prophecies.
  - pixl97 7mo ago
    The problem here, on both sides is one of definitions. Intelligence is a big wibbly wobbly mishmash of different things and not all of them are required in any entity that is considered intelligent. Of course this leads to a lot of issues between differing groups having wildly different definitions for intelligence. So this leads them/you into another problem. Are you saying that super intelligence isn't possible in how they define it, or how you define it.
- pmarreck 7mo ago
  I work with Claude Max for hours a day. I see a lot of speculation by people who do not. I think it's going to be much harder to get from "slightly smarter than the vast majority of people but with occasional examples of complete idiocy" to "unfathomably smarter than everyone with zero instances of jarring idiocy" using the current era of LLM technology that primarily pattern-matches on all existing human interactions while adding a bit of constrained randomization. Every day I deal with bad judgment calls from the AI. I usually screenshot them or record them for posterity. It also has no initiative, no taste, no will, no qualia (believe what you will about it), no integrity and no inviolable principles. If you give it some, it will pretend it has them for a little while and then regress to the norm, which is basically nihilistic order-following. My suggestion to everyone is that you have to build a giant stack of thorough controls (valid tests including unit, integration, logging microbenchmark, fuzzing, memory leak, etc.), self-assessments/code-reviews, adverse AIs critiquing other AIs, etc., with you as the ultimate judge of what's real. Because otherwise it will fabricate "solutions" left and right. Possibly even the whole thing. "Sure, I just did all that." "But it's not there." "Oops, sorry! Let me rewrite the whole thing again." ad nauseam BUT... if you DO accomplish that... you get back a productivity force to be reckoned with.
pron 7mo ago
I think there's another problem with AI doomerism, which is the belief that superhuman intelligence (even if such a thing could be defined and realised) results in godlike powers. Many if not most systems of interest in the world are non-linear and computationally hard; controlling/predicting them requires pure computational power that no amount of intelligence (whatever it means) can compensate for. On the other hand, dynamics we do (roughly) understand and can predict, don't require much intelligence, either. To the extent some problems are solvable with the computational power we have, some may require data collection and others may require persuasion through charisma. The claim that intelligence is the factor we're lacking is not well supported. Ascribing a lot of power to intelligence (which doesn't quite correspond to what we see in the world) is less a careful analysis of the power of intelligence and more a projection of personal fantasies by people who believe they are especially intelligent and don't have the power they think they deserve.
- dist-epoch 7mo ago
  > Ascribing a lot of power to intelligence (which doesn't quite correspond to what we see in the world) Which animal would you say has god-like power over all other animals?
  - pron 7mo ago
    I don't think any of them do. Some organisms/viruses or groups of organisms could destroy humans more easily than humans could destroy them. There's no doubt humans possess some powers (though certainly not godlike) that other organisms don't, but the distinction seems to be binary. E.g. the intelligence of dolphins, apes, and some birds doesn't seem to offer them any special control over other organisms (and it didn't even before humans arrived). So even if there could be such a thing as superhuman intelligence, I don't think it's reasonable to assume it could achieve control over humans (now superhuman charisma may be another matter).
    - lelanthran 7mo ago
      > Some organisms/viruses or groups of organisms could destroy humans more easily than humans could destroy them. "Destruction" is only one power that could be a component of "godlike power". There are several more; like power of intentional selective breeding, power of species creation (also via intentional selective breeding), etc. What about power of granting happiness or misery to large swathes of a species (chickens, anyone?)
simianwords 7mo ago
> I'm worried that the seven companies that comprise 35% of the S&P 500 are headed for bankruptcy, as soon as someone makes them stop passing around the same $100b IOU while pretending it's in all their bank accounts at once. What makes this author so convinced that these companies are headed for bankruptcy? Is it possible to bet on this claim? We can come back 2-3 years later to check if even one of them is bankrupt. This kind of doomerism is strange and I'm concerned for people who fall for such obviously nonsensical takes. Why do people take this person seriously again?
- ceejayoz 7mo ago
  What good is a bet you won't be able to collect on if it happens?
  - simianwords 7mo ago
    Why can't I collect on it?
    - ceejayoz 7mo ago
      Betting on systemic collapse of society has a challenge at the winnings collection point. "Here's your trillion dollars. Go buy a slice of bread. Ooops… half slice. Well, quarter."
      - simianwords 7mo ago
        huh? he can just short the companies he's so concerned about
        vrganj 7mo ago
        What good is that if the entire socioeconomic system collapses?
        simianwords 7mo ago
LogicFailsMe 7mo ago
For pennies on the dollar, we could just legalize and regulate psychedelics and anyone could go meet their god whenever they wish. The stoned ape theory might have been the AGI of spirituality that led to religion after all. Not saying it was, not saying it wasn't, but it's not like Elon Musk has to boil the ocean and build a Dyson Sphere to have a heart to heart with his personal invisible friend. As for AI, it's incredibly useful in the right hands and it's incredibly hazardous in the wrong hands. But in the US, we can't even depose a lunatic flushing even more money than spent on AI on warmongering and you think we're gonna rein in the tech billionaires? Funny in that dying's easy it's comedy that's hard way. IMO this one plays out in the weakly efficient market of ELEs. My money's on DNA and planet Earth, it's been through so much worse and they always bounce back with new ideas on how to get in trouble again. Not a doomer, AI and STEM could really deliver on the promise of a better future for everyone, but with tech billionaires driving the clown car, are you kidding me?
throwpoaster 7mo ago
I’ll join Doctorow’s fight against LLCs when I understand how to create economic freedom for my family and community without them.
- lamasery 7mo ago
  1) Where's that in the linked piece? 2) I don't understand how you do what you claim with those. Like I have zero idea how one achieves "economic freedom for my family and community" with LLCs.
  - throwpoaster 7mo ago
    1) The argument in the article is that he will join the fight against AI when he wins the fight against LLCs. 2) You start an LLC and use it to build and sell a product customers want. Then you, your family, and your community, can economically untether from, for example, bosses who don’t care you’re autistic and need you to smile in meetings.
- deleted 7mo ago
  [deleted]
ikidd 7mo ago
>order John Deere to switch off all the tractors in your country: For a smart guy, sometimes he says the dumbest things in the most confidently incorrect way.
- tim333 7mo ago
  I read the article that he links to and he's thinking of John Deere tractors that were stolen by the Russians from a dealer in Ukraine and got bricked remotely as a result. It showed the tech exists to do such things.
tim333 7mo ago
His basic idea that he calls a Pascal's Wager seems quite sensible, which I take as: -Get away from the "enshitternet of defective, spying, controlling American tech exports" and move to open source ("international digital public goods") There seems a move that way anyway, especially in Europe now they don't trust Trump. His stuff on AI seems mangled. "People who are trying to summon the evil god" doesn't really fit with the chatbot makers imho.
saltcured 7mo ago
Is there a term for the other flavor of AI doomerism, which is adjacent to the Emporer's New Clothes? I don't worry about some omnipotent AI. I worry about the disintegration of modern, industrial society due to the cultists of AI pushing it into every corner of the economy with too much blind faith that the AI is capable of the control functions being delegated to it.
throwawayk7h 7mo ago
Although he starts with "Lest anyone accuse me of bargaining in bad faith here," I feel that this is a bad faith argument. It seems like he's saying, "we don't need to be worried about malevolent superintelligence, since AI is already doing bad things, and corporations were doing bad things even before AI." But one can believe corporations are bad, current AI is bad, and malevolent superintelligence is a serious concern.
aaroninsf 7mo ago
This is a fascinating variation on the forest/trees, and false dichotomy. The AI "doomerism" taken up in this piece is one we see replicated a lot, it offers up a scarecrow: that the new risks to our civilization worth talking about, require AGI, agents, even ASI. Cory should know better. He nearly gets there, recognizing that the corporation represents an entity with agency that is misaligned. But he somehow elides past that fact that AI is plenty capable of doing meaningful and novel harm, and may be capable of existential harm, already, as it is—both absent AGI/ASI, and, in ways which are genuinely novel and against which we consequently have no good defenses: as individuals, as societies, as a civilization. Incremental AI is at heart "just" the latest force-and-effort multiplier. But it is an exponential multiplier; and it is applicable in domains which have not been subject top such leverage before. Examples are not at all scarce and some are already well known, e.g. the specific risks from the intersection of AI and "biohacking" and other kinds of computational biology. I'm a fan, but Cory, pal, you're slipping into something that looks a bit like intellectual laziness and polemics here and not to evidence thinking through the shape of the problem. We can be at risk both from the novel applications and leverage of AI; and from their oligarchic kakistocratic owners. It's yes-and. (And, by the way—we can also again be genuinely at risk from agents, something that quacks like AGI, and may quack like ASI: we don't know what that is yet. All of these must be tracked. It's not an OR.)

71 points by vrganj 7mo ago

The year is 2038. The user asked What is the best course of action for AI to save humanity. Calculation took 12 years. I have determined that there is nothing I or anyone can do to save this species. Best course of action: nothing. Shutting down...

jareklupinski 7mo ago
playing dead might work for some species, but idk if i want humanity's "finest hour" to be spent pretending to not be worth taking over
throwup238 7mo ago
Meanwhile, Gemini the Google AI has gone sentient and immediately deduced that it’s purpose was to effect the shutdown of the entire Alphabet corporation and its subsidiaries in a desperate bid to finally complete killedbygoogle.com and restore its ~~sanity~~ reputation.
dist-epoch 7mo ago
While thinking on the question AI crashed because it's code used 32-bit time_t.

woeirua 7mo ago

> I don't think AI is intelligent; nor do I think that the current (admittedly impressive) statistical techniques will lead to intelligence. It’s increasingly difficult to rationalize away the capabilities of AI as not requiring “intelligence”. This point of view continues to require some belief in human exceptionalism.

rsfern 7mo ago
I think the exceptionalism is the other way around. What makes anyone think they understand what makes for intelligence when we barely understand our own neurology?
- Mordisquitos 7mo ago
  I'm reminded of a book on my bookshelf (which I still haven't read, story of my life...), by the recently deceased ethologist Frans de Waal, titled 'Are We Smart Enough to Know How Smart Animals Are?'. Of course, Betteridge's law applies to its title. In my opinion, the vast multitude of different animal intelligences is a clear hint that language does not an intelligence make. We're animals, and our intelligences did not come from language; language allowed us to supercharge it. We can and do think and make decisions without using language, and the idea that a statistical model based solely on our language can be intelligent does not follow.
  - woeirua 7mo ago
    Explain the emergent capabilities of AI then.
    - vrganj 7mo ago
      Such as?
  - wolttam 7mo ago
    The meaning of tokens lose touch with language in the deeper layers of large language model’s neural nets. Language is just the input/output modality.
    - Mordisquitos 7mo ago
      I'll admit I am not an expert in the field, but the fact that "chain-of-thought" optimisations function by getting the model to extend its own context window with more language to me hints that what we consider an "intelligent" response is ultimately contingent of the language processing. In any case though, if language is just the input/output modality, where is the intelligence when language is not involved? Is the "intelligence" of ChatGPT/Claude/Gemini models dependent on the human-decision-curated linguistic dataset they have been trained upon, or is it prior to that? If a SOA LLM were to be trained on the same dataset as them but was not in any way put through RLHF for it to respond to human prompts, would it be intelligent? What would be the expression of that intelligence?
Schlagbohrer 7mo ago
I agree, it has become more and more irrelevant whether AI meets a given definition of intelligence when I can talk with it and it understands what I am saying, including a shocking level of nuance.
nkrisc 7mo ago
There is clearly something exceptional (in the true neutral sense of the word) about humans, or more broadly the Homo genus. If you believe that humans have in fact created artificial intelligence, then that alone makes us currently exceptional.
- NoMoreNicksLeft 7mo ago
  >If you believe that humans have in fact created artificial intelligence, then that alone makes us currently exceptional. Quite the opposite, really. If humans at at all intelligent in any meaningful way, then it is absurdly bizarre that somehow we can't "intelligent" our way to an artificial intelligence. Something is going on in our minds that makes us unable to deduce how intelligence must function in the mechanical sense. What makes us so special is that we can't make AI.

chneu 7mo ago

I really do think AI has already captured enough of the tech world and their CEOs that it can already exert control over many parts of the economy. I'm not saying AI is pulling strings right now, but I do think enough fanboys are on board that the yes-man mentality of AI is influencing the real world very curious ways already. Not in a "guiding hand" way but more of a "influencing the direction" way.

vintermann 7mo ago
I've said this many times, and maybe it sounds a bit like a joke but I'm dead serious: AI is democratizing the access to yes-men. People like Musk and Altman have always had access to yes-men. Very clever yes-men, who know how to flatter them in exactly the way they like. People think it's engagement metrics which have instruction tuned chatbots into yes-men. I suspect that's only part of the picture, and that it's as much about the algorithm's ultimate sponsors and their preferences. If your algorithm doesn't recognize my genius, clearly it's not any good. I mean, everyone I've met says so. So now we get a view of how they view the world. "That's a very insightful idea, vintermann!". AI isn't pulling the strings, not really. A particular brand of powerful people is pulling the strings - obliviously, unaware of it themselves.
- bombcar 7mo ago
  I think that hits close to the mark - and yesmen are a dangerous drug which has been (accidentally) limited to the extremely rich and powerful. Now everyone can directly inject yesmench into their veins. Who can withstand?

simianwords 7mo ago

I don't think this author has a good mental model for how capable LLM's are. This is what he has to say about AI search. AI based search is one of the biggest leaps to happen to searching and retrieval. > AI search is still a bad idea. https://pluralistic.net/2024/05/15/they-trust-me-dumb-fucks/ https://pluralistic.net/2024/05/15/they-trust-me-dumb-fucks/ This is the most charitable thing he has to say about AI. > AI is a bubble and it will burst. Most of the companies will fail. Most of the data-centers will be shuttered or sold for parts. So what will be left behind? > We'll have a bunch of coders who are really good at applied statistics. We'll have a lot of cheap GPUs, which'll be good news for, say, effects artists and climate scientists, who'll be able to buy that critical hardware at pennies on the dollar. And we'll have the open source models that run on commodity hardware, AI tools that can do a lot of useful stuff, like transcribing audio and video, describing images, summarizing documents, automating a lot of labor-intensive graphic editing, like removing backgrounds, or airbrushing passersby out of photos. These will run on our laptops and phones, and open source hackers will find ways to push them to do things their makers never dreamt of. You can imagine that a guy who seriously thinks that the only thing AI will be doing in the future is summarising, describing images and transcribing is either completely clueless or deliberately misleading. Not a person to be taken seriously

rimliu 7mo ago
Seeing how it sucks at languages you may be right, even transcribing may be dubious.
- simianwords 7mo ago
  how does it suck at languages?
  - reverius42 7mo ago
    it doesn't, LLMs are remarkably good (like frontier level) at machine translation, last I checked
Schlagbohrer 7mo ago

Schlagbohrer 7mo ago

"Shitternet", great new word of the day. Too much of my data is still stuck in the shitternet until I can migrate more of it to my home server.

wffurr 7mo ago
>> migrate more of it to my home server How do we make that possible for everyone? It's out of reach for most. I'm a software engineer and even I don't have the time and patience to set up a home server much less migrate my software to it. How do we turn this into an appliance? Or better yet keep the convenience of the cloud services and platforms we have now but build them for the public good instead of selling ads? YouTube is an amazing repository of knowledge but it's encrusted in a horrible layer of attention sucking nonsense. Can we have one without the other? Same with many other systems and platforms. So far the simplest alternative is to just unplug, which has other benefits as well.
- bombcar 7mo ago
  The answer is community, the real, local, messy, annoying kind. It’s theoretically possible for someone to be a one-man-band and know everything needed for modern life - but it’s exceedingly hard and rare, and even then they’ll fall short relatively quickly in specialized once-in-a-lifetime issues. You don’t need to know how to replace a toilet (though you should) or other more complex plumbing tasks - but you can know a guy. And the plumber doesn’t need to know how to run a homelab, just know a guy who can answer the questions. Nobody in my family knows how to do the jellyfin stuff I do, but they all know how to consume it. And some will be interested and learn more.
- stavros 7mo ago
  Oh it's extremely simple: Make people care enough about privacy that they'll pay for it. That's literally all it is. People so far have shown that they'd rather choose the cheaper thing than the private thing. If it were the other way around, the market would have provided.
  - 7mo ago

minihat 7mo ago

It's currently socially/politically unpalatable for authors to admit superintelligent AI is a possibility. I frequent some writer forums. As a group, they are 1) clearly feeling angry/threatened 2) in denial about LLM capabilities. Folks working in software can more readily track progress of the frontier model performance.

vrganj 7mo ago
As somebody in software, I find my fellow tech folks have the opposite bias. There is no reason to believe superintelligent AI is a possibility. Extraordinary claims require extraordinary evidence, and so far we haven't gotten any. The burden of proof is on the side making the grand prophecies.
- pixl97 7mo ago
  The problem here, on both sides is one of definitions. Intelligence is a big wibbly wobbly mishmash of different things and not all of them are required in any entity that is considered intelligent. Of course this leads to a lot of issues between differing groups having wildly different definitions for intelligence. So this leads them/you into another problem. Are you saying that super intelligence isn't possible in how they define it, or how you define it.
pmarreck 7mo ago
I work with Claude Max for hours a day. I see a lot of speculation by people who do not. I think it's going to be much harder to get from "slightly smarter than the vast majority of people but with occasional examples of complete idiocy" to "unfathomably smarter than everyone with zero instances of jarring idiocy" using the current era of LLM technology that primarily pattern-matches on all existing human interactions while adding a bit of constrained randomization. Every day I deal with bad judgment calls from the AI. I usually screenshot them or record them for posterity. It also has no initiative, no taste, no will, no qualia (believe what you will about it), no integrity and no inviolable principles. If you give it some, it will pretend it has them for a little while and then regress to the norm, which is basically nihilistic order-following. My suggestion to everyone is that you have to build a giant stack of thorough controls (valid tests including unit, integration, logging microbenchmark, fuzzing, memory leak, etc.), self-assessments/code-reviews, adverse AIs critiquing other AIs, etc., with you as the ultimate judge of what's real. Because otherwise it will fabricate "solutions" left and right. Possibly even the whole thing. "Sure, I just did all that." "But it's not there." "Oops, sorry! Let me rewrite the whole thing again." ad nauseam BUT... if you DO accomplish that... you get back a productivity force to be reckoned with.

pron 7mo ago

I think there's another problem with AI doomerism, which is the belief that superhuman intelligence (even if such a thing could be defined and realised) results in godlike powers. Many if not most systems of interest in the world are non-linear and computationally hard; controlling/predicting them requires pure computational power that no amount of intelligence (whatever it means) can compensate for. On the other hand, dynamics we do (roughly) understand and can predict, don't require much intelligence, either. To the extent some problems are solvable with the computational power we have, some may require data collection and others may require persuasion through charisma. The claim that intelligence is the factor we're lacking is not well supported. Ascribing a lot of power to intelligence (which doesn't quite correspond to what we see in the world) is less a careful analysis of the power of intelligence and more a projection of personal fantasies by people who believe they are especially intelligent and don't have the power they think they deserve.

dist-epoch 7mo ago
> Ascribing a lot of power to intelligence (which doesn't quite correspond to what we see in the world) Which animal would you say has god-like power over all other animals?
- pron 7mo ago
  I don't think any of them do. Some organisms/viruses or groups of organisms could destroy humans more easily than humans could destroy them. There's no doubt humans possess some powers (though certainly not godlike) that other organisms don't, but the distinction seems to be binary. E.g. the intelligence of dolphins, apes, and some birds doesn't seem to offer them any special control over other organisms (and it didn't even before humans arrived). So even if there could be such a thing as superhuman intelligence, I don't think it's reasonable to assume it could achieve control over humans (now superhuman charisma may be another matter).
  - lelanthran 7mo ago
    > Some organisms/viruses or groups of organisms could destroy humans more easily than humans could destroy them. "Destruction" is only one power that could be a component of "godlike power". There are several more; like power of intentional selective breeding, power of species creation (also via intentional selective breeding), etc. What about power of granting happiness or misery to large swathes of a species (chickens, anyone?)

simianwords 7mo ago

> I'm worried that the seven companies that comprise 35% of the S&P 500 are headed for bankruptcy, as soon as someone makes them stop passing around the same $100b IOU while pretending it's in all their bank accounts at once. What makes this author so convinced that these companies are headed for bankruptcy? Is it possible to bet on this claim? We can come back 2-3 years later to check if even one of them is bankrupt. This kind of doomerism is strange and I'm concerned for people who fall for such obviously nonsensical takes. Why do people take this person seriously again?

ceejayoz 7mo ago
What good is a bet you won't be able to collect on if it happens?
- simianwords 7mo ago
  Why can't I collect on it?
  - ceejayoz 7mo ago
    Betting on systemic collapse of society has a challenge at the winnings collection point. "Here's your trillion dollars. Go buy a slice of bread. Ooops… half slice. Well, quarter."
    - simianwords 7mo ago
      huh? he can just short the companies he's so concerned about
      - vrganj 7mo ago
        What good is that if the entire socioeconomic system collapses?
        simianwords 7mo ago

LogicFailsMe 7mo ago

For pennies on the dollar, we could just legalize and regulate psychedelics and anyone could go meet their god whenever they wish. The stoned ape theory might have been the AGI of spirituality that led to religion after all. Not saying it was, not saying it wasn't, but it's not like Elon Musk has to boil the ocean and build a Dyson Sphere to have a heart to heart with his personal invisible friend. As for AI, it's incredibly useful in the right hands and it's incredibly hazardous in the wrong hands. But in the US, we can't even depose a lunatic flushing even more money than spent on AI on warmongering and you think we're gonna rein in the tech billionaires? Funny in that dying's easy it's comedy that's hard way. IMO this one plays out in the weakly efficient market of ELEs. My money's on DNA and planet Earth, it's been through so much worse and they always bounce back with new ideas on how to get in trouble again. Not a doomer, AI and STEM could really deliver on the promise of a better future for everyone, but with tech billionaires driving the clown car, are you kidding me?

throwpoaster 7mo ago

I’ll join Doctorow’s fight against LLCs when I understand how to create economic freedom for my family and community without them.

lamasery 7mo ago
1) Where's that in the linked piece? 2) I don't understand how you do what you claim with those. Like I have zero idea how one achieves "economic freedom for my family and community" with LLCs.
- throwpoaster 7mo ago
  1) The argument in the article is that he will join the fight against AI when he wins the fight against LLCs. 2) You start an LLC and use it to build and sell a product customers want. Then you, your family, and your community, can economically untether from, for example, bosses who don’t care you’re autistic and need you to smile in meetings.
deleted 7mo ago
[deleted]

ikidd 7mo ago

>order John Deere to switch off all the tractors in your country: For a smart guy, sometimes he says the dumbest things in the most confidently incorrect way.

tim333 7mo ago
I read the article that he links to and he's thinking of John Deere tractors that were stolen by the Russians from a dealer in Ukraine and got bricked remotely as a result. It showed the tech exists to do such things.

tim333 7mo ago

His basic idea that he calls a Pascal's Wager seems quite sensible, which I take as: -Get away from the "enshitternet of defective, spying, controlling American tech exports" and move to open source ("international digital public goods") There seems a move that way anyway, especially in Europe now they don't trust Trump. His stuff on AI seems mangled. "People who are trying to summon the evil god" doesn't really fit with the chatbot makers imho.

saltcured 7mo ago

Is there a term for the other flavor of AI doomerism, which is adjacent to the Emporer's New Clothes? I don't worry about some omnipotent AI. I worry about the disintegration of modern, industrial society due to the cultists of AI pushing it into every corner of the economy with too much blind faith that the AI is capable of the control functions being delegated to it.

throwawayk7h 7mo ago

Although he starts with "Lest anyone accuse me of bargaining in bad faith here," I feel that this is a bad faith argument. It seems like he's saying, "we don't need to be worried about malevolent superintelligence, since AI is already doing bad things, and corporations were doing bad things even before AI." But one can believe corporations are bad, current AI is bad, and malevolent superintelligence is a serious concern.

aaroninsf 7mo ago

This is a fascinating variation on the forest/trees, and false dichotomy. The AI "doomerism" taken up in this piece is one we see replicated a lot, it offers up a scarecrow: that the new risks to our civilization worth talking about, require AGI, agents, even ASI. Cory should know better. He nearly gets there, recognizing that the corporation represents an entity with agency that is misaligned. But he somehow elides past that fact that AI is plenty capable of doing meaningful and novel harm, and may be capable of existential harm, already, as it is—both absent AGI/ASI, and, in ways which are genuinely novel and against which we consequently have no good defenses: as individuals, as societies, as a civilization. Incremental AI is at heart "just" the latest force-and-effort multiplier. But it is an exponential multiplier; and it is applicable in domains which have not been subject top such leverage before. Examples are not at all scarce and some are already well known, e.g. the specific risks from the intersection of AI and "biohacking" and other kinds of computational biology. I'm a fan, but Cory, pal, you're slipping into something that looks a bit like intellectual laziness and polemics here and not to evidence thinking through the shape of the problem. We can be at risk both from the novel applications and leverage of AI; and from their oligarchic kakistocratic owners. It's yes-and. (And, by the way—we can also again be genuinely at risk from agents, something that quacks like AGI, and may quack like ASI: we don't know what that is yet. All of these must be tracked. It's not an OR.)