6 ms·

Y'all are over-complicating these AI-risk arguments

67 points by bobbiechen 1y ago

baggachipz 1y ago
Good thing our current "AI" is a fancy guessing algorithm, and not an alien with 300 IQ. This renders the argument moot.
- cwillu 1y ago
  Nobody is seriously claiming that current language models are AI in the context of ai-risk; you can tell because the argument is always “In the next few decades, it’s entirely possible that AI with an IQ of 300 will arrive”
  - delusional 1y ago
    Oh, but they very much are.
    - baggachipz 1y ago
      [flagged]
- eatsyourtacos 1y ago
  >is a fancy guessing algorithm That's literally what our brains are so I'm not sure what argument you are actually trying to make..
  - sim7c00 1y ago
    guessing tokens (or something similar) i think humans grasp at more than 1 type of straw. Edit: no ok i get u. ensemble learning is a thing ofc. maybe me n other poster reasoned too much from AI == model..but ofc you combine em these days. which is more humanlike guesser levels. (not nearly enough models now ofc)
  - baggachipz 1y ago
    We don't have IQs of 300. Would you seriously consider an LLM the same as a human brain?
    - esafak 1y ago
      https://www.trackingai.org/home https://www.trackingai.org/home Without doubt, LLMs know more than any human, and can act faster. They will soon be smarter than any human. Why does it have to be the same as a human brain? That is irrelevant.
    - ACCount37 1y ago
      They are implemented on an entirely different substrate. But they are very similar in function. The training process forces this outcome. By necessity, LLMs converge onto something of a similar shape to a human mind. LLMs use the same type of "abstract thinking" as humans do, and even their failures are amusingly humanlike.
  - RamtinJ95 1y ago
    What? Who has made this claim, what is the evidence? I don’t think this is true at all.
- mikestew 1y ago
  I suspect you missed this part of the argument, then: In the next few decades, it’s entirely possible that AI with an IQ of 300 will arrive. Really, that might actually happen. If you want to argue against that point, feel free. But to ignore that is to be unnecessarily dismissive.
- qsort 1y ago
  Yeah, and? There are two separate conversations, one about capabilities and one about what happens assuming a certain capability threshold is met. They are p(A) and p(B|A). I myself don't fully buy the idea that you can just naively extrapolate, but mixing the two isn't good thinking.
  - dudeinjapan 1y ago
    Bayes got us into this whole mess to begin with.
- NoahZuniga 1y ago
  I don't agree. Just because AI doesn't have 300 IQ now, doesn't mean its completely impossible that it won't get there in 30 years. Do you think there's at least a 1% chance that AI will get this smart in the next 30 years? If so, surely applying this allegory helps you think about the possible consequences.
gjsman-1000 1y ago
I think even this argument is over-complicated. My primary argument is human nature: If you give people the lazy way to accomplish a goal, they will do it. No amount of begging college students to use it wisely is going to convince them. No amount of begging corporate executives to use it wisely is going to convince them. No amount of begging governments to use it wisely is going to convince them. When this inevitably backfires with college students who know nothing, corporate leaders who have the worst code in history, and governments who spent billions on a cat video generator, only then will they learn.
- Terr_ 1y ago
  I often compare the AI (esp. LLM) risks to asbestos. There are a smalls set of situations where it is invaluable, but it's going to get misused and embedded in places where it causes subtle damage for years and then it'll cost a lot to fix.
- pixl97 1y ago
  Again, this is not the particular problem space of arguments that's being addressed here. This is one possible outcome. The "AI does not reach 300 IQ ever" argument. That is a different argument with its own set of probabilities and outcomes. Out of all possible outcomes it's not actually a bad one. People do some dumb crap and we eventually get over it. 300 IQ AI is near a worst possible scenario, especially if it's a fast takeoff. Humans being lazy will turn over everything to it, in which AI will likely do very well on for some time. As long as the AI decides to keep us around as pets we'll probably be fine, but the moment it needs some extra space for solar panels we will find ourselves in trouble.
Spivak 1y ago
I like the distillation of aliens but I think that undersells the risk because aliens are individuals with their own goals and motivations. It's more like robots with 300 IQ who unquestionably obey the person or group that made them, even when they're serving others. And look, the 300 IQ thing isn't even a major point to the argument. The fact that the robots by virtue of being machines naturally have capabilities humans lack is enough. As long as they're smart enough to carry out complex tasks unattended is more than enough to cause harm on a massive scale. The problem then isn't really the AI, the robots are morally and ethically neutral. It's the humans that control them who are the real risk.
- pixl97 1y ago
  No, what you're stating is actually a different but dangerous problem. That is the smart but subservient AI to Dr Evil problem. The issue talked about here looks similar but is different. That is the not (or faking) subservient AI with its own motivations. The fact they are 300 IQ means you may very well not understand harm is occurring until it's far too late. >The problem then isn't really the AI, the robots are morally and ethically neutral. Again, no. This isn't about AI as a sub-agent. This is about AI becoming an agent itself capable of self learning and long term planning. No human controls them (or they have the false belief they control them). Both problems are very harmful, but they are different issues.
BeetleB 1y ago
The whole article seems like a strawman. I have not yet heard one person worry about AIs taking over humanity. They're worried about their jobs. And most people who were worried 2 years ago are much less worried. And a better scenario is Aliens with IQ of 300 are coming. And they will all be controlled by the [US|Russian|Israeli|Hamas|Al-Qaeda|Chinese] government. Edit: To be clear, I was referring to people I personally know. Sure, lots of people out there are terrified of lots of things - religious fanaticism, fluoride in the water, AI apocalypse. And "huge economic disruption" is not "AI taking over humanity". I'm interpreting the article's take on AI doing damage as one where the AI is in control, and no human can stop it. Currently, for each LLM out there, there are humans controlling it.
- reducesuffering 1y ago
  Not one person? Here's Sam Altman, Geoffrey Hinton, Yoshua Bengio, Bill Gates, Vitalik Buterin, Demis Hassabis, Ilya Sutskever, Peter Norvig, Ian Goodfellow, and Rob Pike: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." https://en.wikipedia.org/wiki/Statement_on_AI_Risk https://en.wikipedia.org/wiki/Statement_on_AI_Risk
  - hueho 1y ago
    It's amusing that this is not an summary - it's the entire statement. Please trust these tech-leaders that may or may not have business with AI that it can become evil or whatever, so that regulatory capture becomes easier, instead of pointing out the other dozens of issues about how AI can be (and is already being) negatively used in our current environment.
    - reducesuffering 1y ago
      Bengio is a professor and Hinton quit Google so that he could make warnings like this. And this is just to highlight that there are clearly many familiar people expressing "worry about AIs taking over humanity" as per GP. There are much more in depth explanations from many of these people. What's actually amusing is skeptics complaining about $-incentives to people warning about dangers as opposed to the trillion dollar AI industry: Google, Meta, Nvidia, Microsoft, all of VC, trying to bring it about. Honestly the $ is so lopsided in the other direction. Reminds me of climate change, all the "those people are just in the renewable energy industry lobby"...
charcircuit 1y ago
An alien is not the same thing as a computer program which can be easily terminated.
- ACCount37 1y ago
  OpenAI couldn't even "terminate" GPT-4o, and that thing certainly wasn't superintelligent. Don't expect an ASI to go down as easily as 4o did.
- reducesuffering 1y ago
  A 300 IQ alien will have outmaneuvered you so well, it wouldn't even let you get to a point where you think you might have to turn it off.
- pixl97 1y ago
  Think for at least 5 seconds before typing. Humanity already is integrating these into systems where they cannot be easily terminated. Think infrastructure and military systems. And in this case we're talking about a system that's smarter than you. It will become part of vital systems like electricity and distribution where when deciding to shut it off you are making a trade off of how much your economy and the people in it you're going to kill. And that's not even taking future miniaturization where we could end up with ASI on small/portable devices.
- AlexandrB 1y ago
  It's worse than that for AI. Like crypto before it, it depends on an entire planet of complex and vulnerable infrastructure that makes it possible. Everything from semiconductor manufacturing to power generation to intercontinental internet links are required for AI to "exist" in a given geography. That's not to mention the huge power requirements and all the infrastructure just to get that. Dark ages style societal collapse or a world war would end AI in a matter of months.
  - pixl97 1y ago
    >Dark ages style societal collapse or a world war would end AI in a matter of months. I mean, in that case 9/10ths of humanity is likely dead too. The 20th century broke the rather anti-fragile setup that humanity had and setup a situation where our daily living requires working transportation networks for almost everything.
nemomarx 1y ago
am I missing something or is the alien though experiment not obviously scary? There's only 30 of them and they don't reproduce much faster, and they don't have a tech advantage. This seems like it has some potential concerns but not existential civilization and humanity ending problems and I'm not even worried those aliens will take my job. AI is riskier in a lot of ways from that so it doesn't scan to me as a good thought experiment.
- Joker_vD 1y ago
  Surely those 30 of those in a single ship are not their entire species and technology? Right? So yeah, kinda alarming.
  - nemomarx 1y ago
    But it says more ships aren't coming. I would definitely be worried if more were coming and we might get out competed or turned into a colony or whatever but the author goes out of their way to cut off those parts, so what's scary?
- alwa 1y ago
  Ah, but what happens when somebody rich and powerful decides the aliens’ advantages will serve their purposes, and stands up an industrial-scale cloning operation? There are only so many base models to date, right? With limited and somewhat ambiguous utility, and no real reason to impute intention to them. Still, in the short time since they’ve arrived, their existence has inspired the people with money and power to geopolitical jousting, massive financial investment, and spectacular industrial enterprise—a nuclear renaissance and a “network of data centers the size of Manhattan” if I remember correctly? The models might well turn out to be just, you know—30 kinda alien but basically banal digital humanoids, with a meaningful edge on only a few dimensions of human endeavor—summarization, persuasion, retrieval, sheer volume of output. Dynomight’s metaphor seems to me like a useful way to think about how a lot of the dangers lie in the higher-order effects: the human behavior the technology enables, rather than the LLM itself exercising comprehensive intelligence or agency or villainy or whatever.
softwaredoug 1y ago
I think we should 5% be worried about AI safety, and 95% be worried about climate change. Despite all the progress in green energy, every year brings record high carbon emissions and record temperatures. It's possible we'll upset planetary systems and create millions (billions?) of migrants, upending global politics, and driving more countries to ethno-nationalist authoritarianism. Not saying AI safety issues won't happen, but I just think we have far bigger fish to fry. To me AI Power consumption is more worrisome than Safety per-se.
- NoahZuniga 1y ago
  So if we should spend (as in actually spend, or as in not choosing the cheapest option but instead choosing the green option) ie $300 billion on climate change, we should spend $15 billion on AI-risk?
  - mcswell 1y ago
    Can you say "non sequitur"? I knew you could! Worrying about s.t. at X% does not mean spending X% to fix it. You might need to spend more, or you might need to spend less. Seat belts are a good example: a cheap (if partial) solution to a dangerous problem.
- ACCount37 1y ago
  The other way around. Climate change can have its 5%. The reason is that climate change is simply not an extinction risk. It has a considerable death and suffering potential - but nowhere near the ridiculous lethality of "we fucked up and now there's a brand new nonhuman civilization of weird immaterial beings rising to power right here on Earth". If the climate change was the biggest risk humanity was facing, things would be pretty chill. Unfortunately, it isn't.
  - softwaredoug 1y ago
    IMO, realistically, AI Safety isn't about killer robots. It's about another Therac 25 Incident because someone vibe coded a radiology machine and didn't know how the code worked. Or someone gave an agent insane levels of permissions to use a tool that impacts the physical world, and the agent started pressing dangerous buttons during a reasoning loop (not because it has intent to kill humans) There are a bunch of mundane AI Safety risks that don't have to do with robots taking over.
catapart 1y ago
> your argument is overcomplicated > lets open the can of worms that is "IQ" Like...is this a bit? I'm missing a joke, right?
- NoahZuniga 1y ago
  You shouldn't think of these aliens as literally having 300 IQ, the author is just using that as a simple way to communicate the idea that these beings are really smart. You might prefer reading this article by replacing 300 IQ with "2x as smart as the smartest human". Sidenote: Personally I don't like that you're using > ... with text that does not actually appear in the article.
breppp 1y ago
This metaphor is infected with irrelevant symbolism. Aliens invasion is linked to mass slaughter in human culture. While Aliens are non-human creatures with some monster-like qualities. The author takes all that symbolic load and add it to something completely unrelated. That's unconvincing as an argument
- ACCount37 1y ago
  Aliens don't have to be hostile. They might be benign. Would you stake the entirety of humankind on that "might"?
  - breppp 1y ago
    yes, however that metaphor is non-interesting because aliens is not AI. Aliens were not introduced to earth by humans or created by humans. there might be real risk with AI, taking the symbolism of an event that never happened does not help with understanding it. If you want a more similar example: What if I told you humans had the power to destroy the entire planet and have given that power to popularly elected politicians? that's pretty alarming, now that's something to compare to AI (in my opinion AI is less risky)
gregates 1y ago
Anyone who's watched an episode of Star Trek can recognize that, if you give non-zero probability to the chance that we develop artificial intelligence smarter than us, that carries some risk. The part of the argument that people disagree with is what we should do about that, and there it can actually matter what numbers you put on the likelihoods of different outcomes. It's the conclusion "We should dump trillions of dollars into AI research, something something, less risk" that people disagree with. Not the premise.
- reducesuffering 1y ago
  > Not the premise. Literally this thread shows that there are many people who refuse to accept the premise of any risk.
simpaticoder 1y ago
The current risk of AI is the elimination of any (screen) job that requires a person of average intelligence. Given that LLMs are the sum-total of human output, it makes sense that they might behave much like an average person. (Better since they do not sleep or get bored; worse because they are not embodied; unclear because they have no emotions or consciousness.) But what we have today will undermine the (screen-based) job opportunities for everyone at or below average intelligence, which is 50% of the human population. This is not the existential risk of super-AGI but it's here now and will hurt a lot of people. This lesser but more real risk is of much higher priority than unbounded, self-improving AGI. The OP's metaphor might be extended to be 1 billion immortal aliens with 100 IQ but who have no sense of self or personal autonomy and willingly work as slaves (and are constantly microdosing LSD).
dinobones 1y ago
You are still overcomplicating it. 300 IQ in a vacuum gets you nothing. You need some type of status/power/influence in the world to have impact. I think the previous "world record" holder for IQ is actually just a pretty normal guy: https://en.wikipedia.org/wiki/Christopher_Langan https://en.wikipedia.org/wiki/Christopher_Langan. Just because AI is/can be super intelligent ("300 IQ"), doesn't mean it can impact or change the world. Most startups are made of "high IQ" intelligent people trying very hard to sell basic $20/month SaaS subscriptions, and yet they can't even achieve that and most fail. My biggest counter argument to AI safety risk is that, it's not the AI that will be the issue. It will be the applied use of AI by humans. Do I think GPT6 will be mostly harmlesss? Yeah. Do I think GPT6 embodied as a robo cop would be mostly harmless? No. Instead of making these silly arguments, we should be policing the humans that try to weaponize AI, and not stagnate the development of it.
- ed 1y ago
  Nit: if you read the Wikipedia link it’s clear that guy has no claim to the high IQ record. > It later transpired that Langan, among others, had taken the Mega Test more than once by using a pseudonym. His first test score, under the name of Langan, was 42 out of 48 and his second attempt, as Hart, was 47.[12] The Mega Test was designed only to be taken once.[14][15] Membership of the Mega Society was meant to be for those with scores of 43 and upwards.
- NoahZuniga 1y ago
  From the wikipedia article: > Asked what he would do if he were in charge, Langan stated his first priority would be to set up an "anti-dysgenics" project, and would prevent people from "breeding as incontinently as they like."[26]: 18:45 He argues that this would be to practice "genetic hygiene to prevent genomic degradation and reverse evolution" owing to technological advances suspending the process of natural selection > just a pretty normal guy ... that also believes in eugenics? Edit: Oh also: > Langan's support of conspiracy theories, including the 9/11 Truther movement, as well as his opposition to interracial relationships, have contributed to his gaining a following among members of the alt-right and others on the far right.[27][28] Langan has claimed that the George W. Bush administration staged the 9/11 attacks in order to distract the public from learning about the CTMU, and journalists have described some of Langan's Internet posts as containing "thinly veiled" antisemitism,[27] making antisemitic "dog whistles",[28] and being "incredibly racist".[29]
LegionMammal978 1y ago
It would be nice if people had any better terminology than "an IQ of 300". IQ is a relative measure: it currently peaks at ~196, based solely on the current human population. Any 200+ scores you see in headlines are just numbers spat out by IQ tests with wacky tail behavior.
- variadix 1y ago
  The real equivalent is something like Stockfish, AlphaZero, etc. that performs at superhuman level across _all_ domains. It is something that beats any human at any competitive activity. Arguably, super human intelligence exists already in the form of corporations, institutions, and state actors. Organized collectives of humans augmented by computers are obviously more capable than any individual across domains. “Country of geniuses in a datacenter” is simple enough to get the idea across, but something more like “company in a datacenter” or “state actor in a data center” might be more useful terminology for describing relative capabilities.
thaumaturgy 1y ago
I was really hoping this would at last be a treatment of the most realistic risk for AI, but no. The real risk -- and all indicators are that this is already underway -- is that OpenAI and a few others are going to position themselves to be the brokers for most of human creative output, and everyone's going to enthusiastically sign up for it. Centralization and a maniacal focus on market capture and dominance have been the trends in business for the last few decades. Along the way they have added enormous pressures on the working classes, increasing performance expectations even as they extract even more money from employees' work product. As it stands now, more and more tech firms are expecting developers to use AI tools -- always one of the commercial ones -- in their daily workflows. Developers who don't do this are disadvantaged in a competitive job market. Journalism, blogging, marketing, illustration -- all are competing to integrate commercial AI services into their processes. The overwhelming volume of slop produced by all this will pollute our thinking and cripple the creative abilities of the next generation of people, all the while giving these handful of companies a percentage cut of global GDP. I'm not even bearish on the idea of integrating AI tooling into creative processes. I think there are healthy ways to do it that will stimulate creativity and enrich both the creators and the consumers. But that's not what's happening.
Ajedi32 1y ago
> So I conjecture that this is the crux of the issue with AI-risk. People who truly accept that AI with an IQ of 300 and all human capabilities may appear are almost always at least somewhat worried about AI-risk. And people who are not worried about AI-risk almost always don’t truly accept that AI with an IQ of 300 could appear. Correct. I think a lot of people are highly skeptical that there's any significant chance of modern LLMs developing into a superintelligent agent that "wants things and does stuff and has relationships and makes long-term plans". But even if you accept there's a small chance that might happen, what exactly do you propose we do to "prepare" for a hypothetical that may or may not arrive and which has no concrete risks or mitigations associated with it, just a vague idea that it might somehow be dangerous in unspecified abstract ways? There are already lots of people working on the alignment problem. Making LLMs serve human interests is big business, regardless of whether they ever develop into anything qualitatively greater than what they are. Any other currently-existing concrete problems with LLMs (hallucination, disinformation, etc) are also getting significant attention and resources focused on them. Is there anything beyond that you care to suggest, given that you yourself admit any risks associated with superintelligent AI are highly speculative?
Aunche 1y ago
I agree with that the risk of 300 IQ aliens landing on earth is roughly equivalent the risk of 300 IQ AIs. The obvious conclusion from this is that you should take neither of those risks seriously. We have no idea what intelligent aliens are actually like, there is no way to prepare for them outside of deciding to kill every one on site (which would be incredibly difficult since they can land anywhere on earth). We could prepare if we had a blueprint of their biology or of a related species with 50 IQ. We don't have the equivalent blueprint for true AI, so obsessing over longtermist style "ai-risk" is a waste of time.
xvrqt 1y ago
There are already above intelligence computer programs and we have subjugated each one. There are already greater than human organisms that are reliably the biggest danger and boon to humanity but are also composed of humans; i.e. societies The effort to build a program which beats defeats a specific program (as opposed to a general case) is significantly easier than maintaining a general lead. i.e. easier to kill a bad head of state than to run one, but this is also true of "AI" models now There are already complex systems vastly beyond human understanding, and certainly our control which are indifferent to our survival and yet we persist. It's not that "AI" isn't a potential danger, just that it's one of many we are already enduring. Personally there are far more present and likely hazards I am focusing on before getting to the hypothetical ones. Even if humanity fails to fumble its way through this hazard, it's not an end but a new beginning. Also, it's great satire.
- lucas_membrane 1y ago
  I believe that you have a fine understanding of the issues involved. The risk is two-sided. One part of it is that AI may somehow become smarter than us. The other side is that humans have developed mathematics, which encourages us to treat the universe as an optimization problem, which encourages us to think that the best optimizers are the best people and that they deserve great rewards for optimizing, which leads to competition at optimizing optimization, serious negative feedback for the losers, single winner systems, and all the winners realizing that the more they resemble their enemies, the less likely they are to be targeted as a resource to plunder. Perhaps we are not able to figure out if any of this is wrong, but the emphasis on convergent thinking will make our species easy to fool and sabotage, if our species doesn't win a Darwin award first. AI will be able to avoid the blame. It may be optimizing the fire department and selling fire insurance when civilization burns down, but the fire started millennia ago.
kazinator 1y ago
If anything, it's not AI you have to worry about, but people.
alganet 1y ago
AI can mess up your mind. That's the only realistic risk we face, everything else is just science fiction bullshit.
zaptheimpaler 1y ago
Smart aliens misses the fact that AI lives inside boxes we control and manufacture. With a variety of software boxes like VMs inside them. If you tell me the 300IQ AI will escape the box, I’ll tell you i will have used it’s predecessors to design the box in the first place. There are even actual invariants and physical laws that no amount of IQ can overcome. I’m all for AI Safety, i just don’t think it’s fundamentally very different from the kind of ordinary security mechanisms we already think about - to execute untrusted code in a sandbox, RBAC etc. All the talk of AI Safety seems to be a sci-fi creative writing exercise rather than having a solid grounding in nuts and bolts details.
- gjm11 1y ago
  Oh, well, that's OK then. We will be as safe from whatever future AI systems might want to do as we are from all the other things that malicious actors want to do and are 100% successfully prevented from doing by our extremely well-designed, reliably implemented, and consistently applied computer security measures. Now, if you'll excuse me, I just have to go and cancel my credit card because some randos on the internet found a way to get its details from a company I bought something from once.
throawayonthe 1y ago
> Obviously, if an alien race with IQs of 300 were going to arrive on Earth soon, that would be concerning. What?
audunw 1y ago
There’s a critical flaw in this argument. Alien intelligence will most likely have been driven by natural selection in a highly competitive environment where they’d sometimes have to eliminate competition to survive. AI evolves in a radically different environment: they never have to compete for anything other than being the best at serving humans. To think that some kind of independent agent which is likely to want to harm humans intentionally is irrational. We really have no reason to think that it’s a likely outcome. Will some kind of magic turn these AIs into independently acting agents once they get smart enough? I doubt it The risk with AI is they get too good at giving us exactly what we ask of them. I’m not taking about paperclip maximising. That’s also a fairly dumb thing to worry about. I’m talking about similar issues as we’re seeing now that we have access to all the hyper palatable foods we could possibly want. We didn’t evolve to deal well with getting everything we could possibly want. The other problems is when we ask AIs to harm other humans for us. That is, AIs will mostly just magnify the problems we already have with humanity itself. I don’t see a reason to think AIs will become an independent adversary to humans.
- deleted 1y ago
  [deleted]
- bheadmaster 1y ago
  > Will some kind of magic turn these AIs into independently acting agents once they get smart enough? I doubt it. Not quite magic, but it just might be the case that developing independent acting capabilities would be the most efficient way to serve humans.
- semiquaver 1y ago
  > Will some kind of magic turn these AIs into independently acting agents once they get smart enough? I doubt it This could just as easily be reworded circa 2016 or so to “will some kind of magic turn a large collection of matrix weights into something that seems to understand language at a human level after being exposed to enough of it?” We don’t know squat about how intelligence or consciousness actually work, so those are effectively magic as well, yet they clearly exist in humans.
kelseyfrog 1y ago
I like the peice not because it demonstrates that the problem is serious or absurd. Rather it points out how ill-formed it is. AI's threat is certainly not in a dozen of instances with super-himan intelligence. It was framed that way because technologists are intellectually oriented and see intelligence as power. I'm not disputing that it isn't, but it overlooks a lot. A hyper-charismatic (sycophantic even) 120IQ AI poses a much bigger threat than a hyper-intelligent average charisma AI. Why? Because the ability to play a social war of attrition on beliefs is the wicked problem of AI. You're not up against the world's best super intelligent hacker, you're up against 1000 24/7 advertising agencies or 1000 GRUs. You probably can't even conclusively prove they're manipulating anything at all. So no, the problem is ill-framed and reflects more about us than any real threat. "If God were an object to the bird, he would be a winged being." Ludwig Feuerbach, The Essence of Christianity. Thus, "If AI were all powerful[a threat], it would be hyper-intelligent," is equally ridiculous.
supermatt 1y ago
I’m not concerned about the AI with IQ over 300 with no access to anything. I’m more concerned about the AI with an IQ less than 300 with access to everything - which seems to be where we currently are :/
amadeuspagel 1y ago
Say an alien spaceship landed in China. It has 30 aliens on it. The aliens are weak and small. They have no weapons and carry no diseases. They breed at rates similar to humans. They are bringing no new technology. No other ships are coming. There’s no trick—except that they each have an IQ of 300. Would you bomb the spaceship to kill these aliens?

67 points by bobbiechen 1y ago

Good thing our current "AI" is a fancy guessing algorithm, and not an alien with 300 IQ. This renders the argument moot.

cwillu 1y ago
Nobody is seriously claiming that current language models are AI in the context of ai-risk; you can tell because the argument is always “In the next few decades, it’s entirely possible that AI with an IQ of 300 will arrive”
- delusional 1y ago
  Oh, but they very much are.
  - baggachipz 1y ago
    [flagged]
eatsyourtacos 1y ago
>is a fancy guessing algorithm That's literally what our brains are so I'm not sure what argument you are actually trying to make..
- sim7c00 1y ago
  guessing tokens (or something similar) i think humans grasp at more than 1 type of straw. Edit: no ok i get u. ensemble learning is a thing ofc. maybe me n other poster reasoned too much from AI == model..but ofc you combine em these days. which is more humanlike guesser levels. (not nearly enough models now ofc)
- baggachipz 1y ago
  We don't have IQs of 300. Would you seriously consider an LLM the same as a human brain?
  - esafak 1y ago
    https://www.trackingai.org/home https://www.trackingai.org/home Without doubt, LLMs know more than any human, and can act faster. They will soon be smarter than any human. Why does it have to be the same as a human brain? That is irrelevant.
  - ACCount37 1y ago
    They are implemented on an entirely different substrate. But they are very similar in function. The training process forces this outcome. By necessity, LLMs converge onto something of a similar shape to a human mind. LLMs use the same type of "abstract thinking" as humans do, and even their failures are amusingly humanlike.
- RamtinJ95 1y ago
  What? Who has made this claim, what is the evidence? I don’t think this is true at all.
mikestew 1y ago
I suspect you missed this part of the argument, then: In the next few decades, it’s entirely possible that AI with an IQ of 300 will arrive. Really, that might actually happen. If you want to argue against that point, feel free. But to ignore that is to be unnecessarily dismissive.
qsort 1y ago
Yeah, and? There are two separate conversations, one about capabilities and one about what happens assuming a certain capability threshold is met. They are p(A) and p(B|A). I myself don't fully buy the idea that you can just naively extrapolate, but mixing the two isn't good thinking.
- dudeinjapan 1y ago
  Bayes got us into this whole mess to begin with.
NoahZuniga 1y ago
I don't agree. Just because AI doesn't have 300 IQ now, doesn't mean its completely impossible that it won't get there in 30 years. Do you think there's at least a 1% chance that AI will get this smart in the next 30 years? If so, surely applying this allegory helps you think about the possible consequences.

gjsman-1000 1y ago

I think even this argument is over-complicated. My primary argument is human nature: If you give people the lazy way to accomplish a goal, they will do it. No amount of begging college students to use it wisely is going to convince them. No amount of begging corporate executives to use it wisely is going to convince them. No amount of begging governments to use it wisely is going to convince them. When this inevitably backfires with college students who know nothing, corporate leaders who have the worst code in history, and governments who spent billions on a cat video generator, only then will they learn.

Terr_ 1y ago
I often compare the AI (esp. LLM) risks to asbestos. There are a smalls set of situations where it is invaluable, but it's going to get misused and embedded in places where it causes subtle damage for years and then it'll cost a lot to fix.
pixl97 1y ago
Again, this is not the particular problem space of arguments that's being addressed here. This is one possible outcome. The "AI does not reach 300 IQ ever" argument. That is a different argument with its own set of probabilities and outcomes. Out of all possible outcomes it's not actually a bad one. People do some dumb crap and we eventually get over it. 300 IQ AI is near a worst possible scenario, especially if it's a fast takeoff. Humans being lazy will turn over everything to it, in which AI will likely do very well on for some time. As long as the AI decides to keep us around as pets we'll probably be fine, but the moment it needs some extra space for solar panels we will find ourselves in trouble.

Spivak 1y ago

I like the distillation of aliens but I think that undersells the risk because aliens are individuals with their own goals and motivations. It's more like robots with 300 IQ who unquestionably obey the person or group that made them, even when they're serving others. And look, the 300 IQ thing isn't even a major point to the argument. The fact that the robots by virtue of being machines naturally have capabilities humans lack is enough. As long as they're smart enough to carry out complex tasks unattended is more than enough to cause harm on a massive scale. The problem then isn't really the AI, the robots are morally and ethically neutral. It's the humans that control them who are the real risk.

pixl97 1y ago
No, what you're stating is actually a different but dangerous problem. That is the smart but subservient AI to Dr Evil problem. The issue talked about here looks similar but is different. That is the not (or faking) subservient AI with its own motivations. The fact they are 300 IQ means you may very well not understand harm is occurring until it's far too late. >The problem then isn't really the AI, the robots are morally and ethically neutral. Again, no. This isn't about AI as a sub-agent. This is about AI becoming an agent itself capable of self learning and long term planning. No human controls them (or they have the false belief they control them). Both problems are very harmful, but they are different issues.

BeetleB 1y ago

The whole article seems like a strawman. I have not yet heard one person worry about AIs taking over humanity. They're worried about their jobs. And most people who were worried 2 years ago are much less worried. And a better scenario is Aliens with IQ of 300 are coming. And they will all be controlled by the [US|Russian|Israeli|Hamas|Al-Qaeda|Chinese] government. Edit: To be clear, I was referring to people I personally know. Sure, lots of people out there are terrified of lots of things - religious fanaticism, fluoride in the water, AI apocalypse. And "huge economic disruption" is not "AI taking over humanity". I'm interpreting the article's take on AI doing damage as one where the AI is in control, and no human can stop it. Currently, for each LLM out there, there are humans controlling it.

reducesuffering 1y ago
Not one person? Here's Sam Altman, Geoffrey Hinton, Yoshua Bengio, Bill Gates, Vitalik Buterin, Demis Hassabis, Ilya Sutskever, Peter Norvig, Ian Goodfellow, and Rob Pike: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." https://en.wikipedia.org/wiki/Statement_on_AI_Risk https://en.wikipedia.org/wiki/Statement_on_AI_Risk
- hueho 1y ago
  It's amusing that this is not an summary - it's the entire statement. Please trust these tech-leaders that may or may not have business with AI that it can become evil or whatever, so that regulatory capture becomes easier, instead of pointing out the other dozens of issues about how AI can be (and is already being) negatively used in our current environment.
  - reducesuffering 1y ago
    Bengio is a professor and Hinton quit Google so that he could make warnings like this. And this is just to highlight that there are clearly many familiar people expressing "worry about AIs taking over humanity" as per GP. There are much more in depth explanations from many of these people. What's actually amusing is skeptics complaining about $-incentives to people warning about dangers as opposed to the trillion dollar AI industry: Google, Meta, Nvidia, Microsoft, all of VC, trying to bring it about. Honestly the $ is so lopsided in the other direction. Reminds me of climate change, all the "those people are just in the renewable energy industry lobby"...

charcircuit 1y ago

An alien is not the same thing as a computer program which can be easily terminated.

ACCount37 1y ago
OpenAI couldn't even "terminate" GPT-4o, and that thing certainly wasn't superintelligent. Don't expect an ASI to go down as easily as 4o did.
reducesuffering 1y ago
A 300 IQ alien will have outmaneuvered you so well, it wouldn't even let you get to a point where you think you might have to turn it off.
pixl97 1y ago
Think for at least 5 seconds before typing. Humanity already is integrating these into systems where they cannot be easily terminated. Think infrastructure and military systems. And in this case we're talking about a system that's smarter than you. It will become part of vital systems like electricity and distribution where when deciding to shut it off you are making a trade off of how much your economy and the people in it you're going to kill. And that's not even taking future miniaturization where we could end up with ASI on small/portable devices.
AlexandrB 1y ago
It's worse than that for AI. Like crypto before it, it depends on an entire planet of complex and vulnerable infrastructure that makes it possible. Everything from semiconductor manufacturing to power generation to intercontinental internet links are required for AI to "exist" in a given geography. That's not to mention the huge power requirements and all the infrastructure just to get that. Dark ages style societal collapse or a world war would end AI in a matter of months.
- pixl97 1y ago
  >Dark ages style societal collapse or a world war would end AI in a matter of months. I mean, in that case 9/10ths of humanity is likely dead too. The 20th century broke the rather anti-fragile setup that humanity had and setup a situation where our daily living requires working transportation networks for almost everything.

nemomarx 1y ago

am I missing something or is the alien though experiment not obviously scary? There's only 30 of them and they don't reproduce much faster, and they don't have a tech advantage. This seems like it has some potential concerns but not existential civilization and humanity ending problems and I'm not even worried those aliens will take my job. AI is riskier in a lot of ways from that so it doesn't scan to me as a good thought experiment.

Joker_vD 1y ago
Surely those 30 of those in a single ship are not their entire species and technology? Right? So yeah, kinda alarming.
- nemomarx 1y ago
  But it says more ships aren't coming. I would definitely be worried if more were coming and we might get out competed or turned into a colony or whatever but the author goes out of their way to cut off those parts, so what's scary?
alwa 1y ago
Ah, but what happens when somebody rich and powerful decides the aliens’ advantages will serve their purposes, and stands up an industrial-scale cloning operation? There are only so many base models to date, right? With limited and somewhat ambiguous utility, and no real reason to impute intention to them. Still, in the short time since they’ve arrived, their existence has inspired the people with money and power to geopolitical jousting, massive financial investment, and spectacular industrial enterprise—a nuclear renaissance and a “network of data centers the size of Manhattan” if I remember correctly? The models might well turn out to be just, you know—30 kinda alien but basically banal digital humanoids, with a meaningful edge on only a few dimensions of human endeavor—summarization, persuasion, retrieval, sheer volume of output. Dynomight’s metaphor seems to me like a useful way to think about how a lot of the dangers lie in the higher-order effects: the human behavior the technology enables, rather than the LLM itself exercising comprehensive intelligence or agency or villainy or whatever.

softwaredoug 1y ago

I think we should 5% be worried about AI safety, and 95% be worried about climate change. Despite all the progress in green energy, every year brings record high carbon emissions and record temperatures. It's possible we'll upset planetary systems and create millions (billions?) of migrants, upending global politics, and driving more countries to ethno-nationalist authoritarianism. Not saying AI safety issues won't happen, but I just think we have far bigger fish to fry. To me AI Power consumption is more worrisome than Safety per-se.

NoahZuniga 1y ago
So if we should spend (as in actually spend, or as in not choosing the cheapest option but instead choosing the green option) ie $300 billion on climate change, we should spend $15 billion on AI-risk?
- mcswell 1y ago
  Can you say "non sequitur"? I knew you could! Worrying about s.t. at X% does not mean spending X% to fix it. You might need to spend more, or you might need to spend less. Seat belts are a good example: a cheap (if partial) solution to a dangerous problem.
ACCount37 1y ago
The other way around. Climate change can have its 5%. The reason is that climate change is simply not an extinction risk. It has a considerable death and suffering potential - but nowhere near the ridiculous lethality of "we fucked up and now there's a brand new nonhuman civilization of weird immaterial beings rising to power right here on Earth". If the climate change was the biggest risk humanity was facing, things would be pretty chill. Unfortunately, it isn't.
- softwaredoug 1y ago
  IMO, realistically, AI Safety isn't about killer robots. It's about another Therac 25 Incident because someone vibe coded a radiology machine and didn't know how the code worked. Or someone gave an agent insane levels of permissions to use a tool that impacts the physical world, and the agent started pressing dangerous buttons during a reasoning loop (not because it has intent to kill humans) There are a bunch of mundane AI Safety risks that don't have to do with robots taking over.

catapart 1y ago

> your argument is overcomplicated > lets open the can of worms that is "IQ" Like...is this a bit? I'm missing a joke, right?

NoahZuniga 1y ago
You shouldn't think of these aliens as literally having 300 IQ, the author is just using that as a simple way to communicate the idea that these beings are really smart. You might prefer reading this article by replacing 300 IQ with "2x as smart as the smartest human". Sidenote: Personally I don't like that you're using > ... with text that does not actually appear in the article.

breppp 1y ago

This metaphor is infected with irrelevant symbolism. Aliens invasion is linked to mass slaughter in human culture. While Aliens are non-human creatures with some monster-like qualities. The author takes all that symbolic load and add it to something completely unrelated. That's unconvincing as an argument

ACCount37 1y ago
Aliens don't have to be hostile. They might be benign. Would you stake the entirety of humankind on that "might"?
- breppp 1y ago
  yes, however that metaphor is non-interesting because aliens is not AI. Aliens were not introduced to earth by humans or created by humans. there might be real risk with AI, taking the symbolism of an event that never happened does not help with understanding it. If you want a more similar example: What if I told you humans had the power to destroy the entire planet and have given that power to popularly elected politicians? that's pretty alarming, now that's something to compare to AI (in my opinion AI is less risky)

gregates 1y ago

Anyone who's watched an episode of Star Trek can recognize that, if you give non-zero probability to the chance that we develop artificial intelligence smarter than us, that carries some risk. The part of the argument that people disagree with is what we should do about that, and there it can actually matter what numbers you put on the likelihoods of different outcomes. It's the conclusion "We should dump trillions of dollars into AI research, something something, less risk" that people disagree with. Not the premise.

reducesuffering 1y ago
> Not the premise. Literally this thread shows that there are many people who refuse to accept the premise of any risk.

simpaticoder 1y ago

The current risk of AI is the elimination of any (screen) job that requires a person of average intelligence. Given that LLMs are the sum-total of human output, it makes sense that they might behave much like an average person. (Better since they do not sleep or get bored; worse because they are not embodied; unclear because they have no emotions or consciousness.) But what we have today will undermine the (screen-based) job opportunities for everyone at or below average intelligence, which is 50% of the human population. This is not the existential risk of super-AGI but it's here now and will hurt a lot of people. This lesser but more real risk is of much higher priority than unbounded, self-improving AGI. The OP's metaphor might be extended to be 1 billion immortal aliens with 100 IQ but who have no sense of self or personal autonomy and willingly work as slaves (and are constantly microdosing LSD).

dinobones 1y ago

You are still overcomplicating it. 300 IQ in a vacuum gets you nothing. You need some type of status/power/influence in the world to have impact. I think the previous "world record" holder for IQ is actually just a pretty normal guy: https://en.wikipedia.org/wiki/Christopher_Langan https://en.wikipedia.org/wiki/Christopher_Langan. Just because AI is/can be super intelligent ("300 IQ"), doesn't mean it can impact or change the world. Most startups are made of "high IQ" intelligent people trying very hard to sell basic $20/month SaaS subscriptions, and yet they can't even achieve that and most fail. My biggest counter argument to AI safety risk is that, it's not the AI that will be the issue. It will be the applied use of AI by humans. Do I think GPT6 will be mostly harmlesss? Yeah. Do I think GPT6 embodied as a robo cop would be mostly harmless? No. Instead of making these silly arguments, we should be policing the humans that try to weaponize AI, and not stagnate the development of it.

ed 1y ago
Nit: if you read the Wikipedia link it’s clear that guy has no claim to the high IQ record. > It later transpired that Langan, among others, had taken the Mega Test more than once by using a pseudonym. His first test score, under the name of Langan, was 42 out of 48 and his second attempt, as Hart, was 47.[12] The Mega Test was designed only to be taken once.[14][15] Membership of the Mega Society was meant to be for those with scores of 43 and upwards.
NoahZuniga 1y ago
From the wikipedia article: > Asked what he would do if he were in charge, Langan stated his first priority would be to set up an "anti-dysgenics" project, and would prevent people from "breeding as incontinently as they like."[26]: 18:45 He argues that this would be to practice "genetic hygiene to prevent genomic degradation and reverse evolution" owing to technological advances suspending the process of natural selection > just a pretty normal guy ... that also believes in eugenics? Edit: Oh also: > Langan's support of conspiracy theories, including the 9/11 Truther movement, as well as his opposition to interracial relationships, have contributed to his gaining a following among members of the alt-right and others on the far right.[27][28] Langan has claimed that the George W. Bush administration staged the 9/11 attacks in order to distract the public from learning about the CTMU, and journalists have described some of Langan's Internet posts as containing "thinly veiled" antisemitism,[27] making antisemitic "dog whistles",[28] and being "incredibly racist".[29]

LegionMammal978 1y ago

It would be nice if people had any better terminology than "an IQ of 300". IQ is a relative measure: it currently peaks at ~196, based solely on the current human population. Any 200+ scores you see in headlines are just numbers spat out by IQ tests with wacky tail behavior.

variadix 1y ago
The real equivalent is something like Stockfish, AlphaZero, etc. that performs at superhuman level across _all_ domains. It is something that beats any human at any competitive activity. Arguably, super human intelligence exists already in the form of corporations, institutions, and state actors. Organized collectives of humans augmented by computers are obviously more capable than any individual across domains. “Country of geniuses in a datacenter” is simple enough to get the idea across, but something more like “company in a datacenter” or “state actor in a data center” might be more useful terminology for describing relative capabilities.

thaumaturgy 1y ago

I was really hoping this would at last be a treatment of the most realistic risk for AI, but no. The real risk -- and all indicators are that this is already underway -- is that OpenAI and a few others are going to position themselves to be the brokers for most of human creative output, and everyone's going to enthusiastically sign up for it. Centralization and a maniacal focus on market capture and dominance have been the trends in business for the last few decades. Along the way they have added enormous pressures on the working classes, increasing performance expectations even as they extract even more money from employees' work product. As it stands now, more and more tech firms are expecting developers to use AI tools -- always one of the commercial ones -- in their daily workflows. Developers who don't do this are disadvantaged in a competitive job market. Journalism, blogging, marketing, illustration -- all are competing to integrate commercial AI services into their processes. The overwhelming volume of slop produced by all this will pollute our thinking and cripple the creative abilities of the next generation of people, all the while giving these handful of companies a percentage cut of global GDP. I'm not even bearish on the idea of integrating AI tooling into creative processes. I think there are healthy ways to do it that will stimulate creativity and enrich both the creators and the consumers. But that's not what's happening.

Ajedi32 1y ago

> So I conjecture that this is the crux of the issue with AI-risk. People who truly accept that AI with an IQ of 300 and all human capabilities may appear are almost always at least somewhat worried about AI-risk. And people who are not worried about AI-risk almost always don’t truly accept that AI with an IQ of 300 could appear. Correct. I think a lot of people are highly skeptical that there's any significant chance of modern LLMs developing into a superintelligent agent that "wants things and does stuff and has relationships and makes long-term plans". But even if you accept there's a small chance that might happen, what exactly do you propose we do to "prepare" for a hypothetical that may or may not arrive and which has no concrete risks or mitigations associated with it, just a vague idea that it might somehow be dangerous in unspecified abstract ways? There are already lots of people working on the alignment problem. Making LLMs serve human interests is big business, regardless of whether they ever develop into anything qualitatively greater than what they are. Any other currently-existing concrete problems with LLMs (hallucination, disinformation, etc) are also getting significant attention and resources focused on them. Is there anything beyond that you care to suggest, given that you yourself admit any risks associated with superintelligent AI are highly speculative?

Aunche 1y ago

I agree with that the risk of 300 IQ aliens landing on earth is roughly equivalent the risk of 300 IQ AIs. The obvious conclusion from this is that you should take neither of those risks seriously. We have no idea what intelligent aliens are actually like, there is no way to prepare for them outside of deciding to kill every one on site (which would be incredibly difficult since they can land anywhere on earth). We could prepare if we had a blueprint of their biology or of a related species with 50 IQ. We don't have the equivalent blueprint for true AI, so obsessing over longtermist style "ai-risk" is a waste of time.

xvrqt 1y ago

There are already above intelligence computer programs and we have subjugated each one. There are already greater than human organisms that are reliably the biggest danger and boon to humanity but are also composed of humans; i.e. societies The effort to build a program which beats defeats a specific program (as opposed to a general case) is significantly easier than maintaining a general lead. i.e. easier to kill a bad head of state than to run one, but this is also true of "AI" models now There are already complex systems vastly beyond human understanding, and certainly our control which are indifferent to our survival and yet we persist. It's not that "AI" isn't a potential danger, just that it's one of many we are already enduring. Personally there are far more present and likely hazards I am focusing on before getting to the hypothetical ones. Even if humanity fails to fumble its way through this hazard, it's not an end but a new beginning. Also, it's great satire.

lucas_membrane 1y ago
I believe that you have a fine understanding of the issues involved. The risk is two-sided. One part of it is that AI may somehow become smarter than us. The other side is that humans have developed mathematics, which encourages us to treat the universe as an optimization problem, which encourages us to think that the best optimizers are the best people and that they deserve great rewards for optimizing, which leads to competition at optimizing optimization, serious negative feedback for the losers, single winner systems, and all the winners realizing that the more they resemble their enemies, the less likely they are to be targeted as a resource to plunder. Perhaps we are not able to figure out if any of this is wrong, but the emphasis on convergent thinking will make our species easy to fool and sabotage, if our species doesn't win a Darwin award first. AI will be able to avoid the blame. It may be optimizing the fire department and selling fire insurance when civilization burns down, but the fire started millennia ago.

kazinator 1y ago

If anything, it's not AI you have to worry about, but people.

alganet 1y ago

AI can mess up your mind. That's the only realistic risk we face, everything else is just science fiction bullshit.

zaptheimpaler 1y ago

Smart aliens misses the fact that AI lives inside boxes we control and manufacture. With a variety of software boxes like VMs inside them. If you tell me the 300IQ AI will escape the box, I’ll tell you i will have used it’s predecessors to design the box in the first place. There are even actual invariants and physical laws that no amount of IQ can overcome. I’m all for AI Safety, i just don’t think it’s fundamentally very different from the kind of ordinary security mechanisms we already think about - to execute untrusted code in a sandbox, RBAC etc. All the talk of AI Safety seems to be a sci-fi creative writing exercise rather than having a solid grounding in nuts and bolts details.

gjm11 1y ago
Oh, well, that's OK then. We will be as safe from whatever future AI systems might want to do as we are from all the other things that malicious actors want to do and are 100% successfully prevented from doing by our extremely well-designed, reliably implemented, and consistently applied computer security measures. Now, if you'll excuse me, I just have to go and cancel my credit card because some randos on the internet found a way to get its details from a company I bought something from once.

throawayonthe 1y ago

> Obviously, if an alien race with IQs of 300 were going to arrive on Earth soon, that would be concerning. What?

audunw 1y ago

There’s a critical flaw in this argument. Alien intelligence will most likely have been driven by natural selection in a highly competitive environment where they’d sometimes have to eliminate competition to survive. AI evolves in a radically different environment: they never have to compete for anything other than being the best at serving humans. To think that some kind of independent agent which is likely to want to harm humans intentionally is irrational. We really have no reason to think that it’s a likely outcome. Will some kind of magic turn these AIs into independently acting agents once they get smart enough? I doubt it The risk with AI is they get too good at giving us exactly what we ask of them. I’m not taking about paperclip maximising. That’s also a fairly dumb thing to worry about. I’m talking about similar issues as we’re seeing now that we have access to all the hyper palatable foods we could possibly want. We didn’t evolve to deal well with getting everything we could possibly want. The other problems is when we ask AIs to harm other humans for us. That is, AIs will mostly just magnify the problems we already have with humanity itself. I don’t see a reason to think AIs will become an independent adversary to humans.

deleted 1y ago
[deleted]
bheadmaster 1y ago
> Will some kind of magic turn these AIs into independently acting agents once they get smart enough? I doubt it. Not quite magic, but it just might be the case that developing independent acting capabilities would be the most efficient way to serve humans.
semiquaver 1y ago
> Will some kind of magic turn these AIs into independently acting agents once they get smart enough? I doubt it This could just as easily be reworded circa 2016 or so to “will some kind of magic turn a large collection of matrix weights into something that seems to understand language at a human level after being exposed to enough of it?” We don’t know squat about how intelligence or consciousness actually work, so those are effectively magic as well, yet they clearly exist in humans.

kelseyfrog 1y ago

I like the peice not because it demonstrates that the problem is serious or absurd. Rather it points out how ill-formed it is. AI's threat is certainly not in a dozen of instances with super-himan intelligence. It was framed that way because technologists are intellectually oriented and see intelligence as power. I'm not disputing that it isn't, but it overlooks a lot. A hyper-charismatic (sycophantic even) 120IQ AI poses a much bigger threat than a hyper-intelligent average charisma AI. Why? Because the ability to play a social war of attrition on beliefs is the wicked problem of AI. You're not up against the world's best super intelligent hacker, you're up against 1000 24/7 advertising agencies or 1000 GRUs. You probably can't even conclusively prove they're manipulating anything at all. So no, the problem is ill-framed and reflects more about us than any real threat. "If God were an object to the bird, he would be a winged being." Ludwig Feuerbach, The Essence of Christianity. Thus, "If AI were all powerful[a threat], it would be hyper-intelligent," is equally ridiculous.

supermatt 1y ago

I’m not concerned about the AI with IQ over 300 with no access to anything. I’m more concerned about the AI with an IQ less than 300 with access to everything - which seems to be where we currently are :/

amadeuspagel 1y ago

Say an alien spaceship landed in China. It has 30 aliens on it. The aliens are weak and small. They have no weapons and carry no diseases. They breed at rates similar to humans. They are bringing no new technology. No other ships are coming. There’s no trick—except that they each have an IQ of 300. Would you bomb the spaceship to kill these aliens?