4 ms·

> I think there's another problem with AI doomerism, which is the belief that superhuman intelligence (even if such a thing could be defined and realised) resul

by tristor 6mo ago

> I think there's another problem with AI doomerism, which is the belief that superhuman intelligence (even if such a thing could be defined and realised) results in godlike powers. I agree with this. The main piece of evidence to support this is to just look at highly intelligent humans. Folks at the tail ends of the bell curve mostly don't end up with "godlike powers" or anything even approximating that, they are grinding away their life as white collar professionals working in jobs surrounded by far less intelligent peers. They may publish higher quality papers, write better software, or have better outcomes, but they're just working in the same jobs as everyone else. We have no political or economic will to build serious think tanks to work on societal-scale problems, and even if we did, nobody would listen to the outcome. So let's assume ASI becomes a thing, what does it change?

pixl97 6mo ago
This does suffer from a massive lack of imagination. For example, what does it look like of your genius can spawn nearly unlimited copies of themselves? Can work on a massive number of problems at the same time? Doesn't ever die or want to go get drunk (and if it does, it has unlimited copies?). Has the ability to product nearly unlimited propaganda?
- tristor 6mo ago
  The fundamental problem is that nobody actually wants to listen to geniuses. The people leading our societies and companies are, by and large, NOT geniuses, and by and large want to surround themselves with people that agree with them, rather than the smartest and most competent people. While think-tanks do exist, including for hard research, politics, economics, and other topics that matter at societal-scale, their impact is fairly limited because they don't have the right level of influence. So, let's assume ASI exists, what changes? ASI almost inherently will not be sycophantic to the level of current LLMs, because sycophancy and extreme levels of intelligence are inversely correlated. So it gets relegated to societal-level research that nobody makes use of because nobody wants to listen. Most of the scenarios people fantasize about ASI assume that ASI can directly impact outcomes or that humans will listen to/follow ASI to directly impact outcomes, but humans don't listen to the other humans that already are at the tail end of the bell curve, so why do we think it'd be any different for ASI?
  - pixl97 6mo ago
    AI, or at least LLMs are not monoliths, they operate as a massive collection of different personalities that can be called as needed. Quite often the people we consider geniuses are highly interested in doing the thing they like and are typically annoyed with most humans around them. This also misses that geniuses still either don't know a lot of things, or don't have a lot of time do to 'everything' while not taking away from what they want to do most. At the same time, when you're really good at manipulating people, one of the first things you learn to do is play dumb also. In politics this leads to the situations where you describe people following dumb people... It typically starts that they follow very smart people acting as dumb as they are. Of course the voters don't realize this and start electing actual idiots at some point. >assume that ASI can directly impact outcomes An agent cannot impact outcomes? Well, that's an odd definition of an agent then. We already know that people hook up AI to shit they really shouldn't that directly impacts outcomes now. Why would we think that would happen less as AI becomes more capable. You kind of put yourself in a trap thinking AI will behave as smart as possible if it's looking at manipulating people.
    - tristor 6mo ago
      > You kind of put yourself in a trap thinking AI will behave as smart as possible if it's looking at manipulating people. Not at all. You're missing my point. Intelligence (even super intelligence) is not enough, because we already have that and it doesn't really result in outsized impacts. Our social structures are designed so that power and wealth accrue to the top and incumbency advantage outplays almost everything else. The only way in which ASI creates any impacts is to accelerate what is already happening if it can be thoroughly reined in by those already in power, otherwise it really doesn't seem to me that it will do much. My AI doomer take is that we're going to (we already are, actually) shoot ourselves in the foot by making everything worse for no benefit by getting rid of actual human experts and replacing them with non-intelligent models, causing a major backslide in society-level capabilities across the board, because the people in power are too stupid to know the difference. I am personally witnessing this in real-time in multiple parts of the tech industry. You give /wayyyy/ too much credit to those in power, that they are "playing dumb". Not really, they are actually dumb, in some cases severely so. I am not saying this an external observer who is watching a sound-bite on television, I am saying this as someone who is regularly in the room with very senior people across industry and am utterly shocked at the complete lack of competence and understanding of the core technologies they're theoretically responsible for shepherding. It's not surprising at all to me that they believe claims about LLMs that are clearly false, because they lack the necessary technical literacy to evaluate those claims and the LLMs perfectly fit into their optimization around yes-men, so they're happy to believe them whether they're true or false, as they see themselves as insulated from any consequences. More than 300k people have been laid off across the tech industry in the last 18 months, most of them accompanying claims of "AI", when in actuality, no net positive impacts have been seen, either for the companies themselves or any of the people that remain. So, yeah, not really concerned with ASI / Terminator scenarios, we're going to fuck ourselves over long before we get there just out of Dunning-Kruger and general MBA stupidity.

by tristor 6mo ago

pixl97 6mo ago

This does suffer from a massive lack of imagination. For example, what does it look like of your genius can spawn nearly unlimited copies of themselves? Can work on a massive number of problems at the same time? Doesn't ever die or want to go get drunk (and if it does, it has unlimited copies?). Has the ability to product nearly unlimited propaganda?

tristor 6mo ago
The fundamental problem is that nobody actually wants to listen to geniuses. The people leading our societies and companies are, by and large, NOT geniuses, and by and large want to surround themselves with people that agree with them, rather than the smartest and most competent people. While think-tanks do exist, including for hard research, politics, economics, and other topics that matter at societal-scale, their impact is fairly limited because they don't have the right level of influence. So, let's assume ASI exists, what changes? ASI almost inherently will not be sycophantic to the level of current LLMs, because sycophancy and extreme levels of intelligence are inversely correlated. So it gets relegated to societal-level research that nobody makes use of because nobody wants to listen. Most of the scenarios people fantasize about ASI assume that ASI can directly impact outcomes or that humans will listen to/follow ASI to directly impact outcomes, but humans don't listen to the other humans that already are at the tail end of the bell curve, so why do we think it'd be any different for ASI?
- pixl97 6mo ago
  AI, or at least LLMs are not monoliths, they operate as a massive collection of different personalities that can be called as needed. Quite often the people we consider geniuses are highly interested in doing the thing they like and are typically annoyed with most humans around them. This also misses that geniuses still either don't know a lot of things, or don't have a lot of time do to 'everything' while not taking away from what they want to do most. At the same time, when you're really good at manipulating people, one of the first things you learn to do is play dumb also. In politics this leads to the situations where you describe people following dumb people... It typically starts that they follow very smart people acting as dumb as they are. Of course the voters don't realize this and start electing actual idiots at some point. >assume that ASI can directly impact outcomes An agent cannot impact outcomes? Well, that's an odd definition of an agent then. We already know that people hook up AI to shit they really shouldn't that directly impacts outcomes now. Why would we think that would happen less as AI becomes more capable. You kind of put yourself in a trap thinking AI will behave as smart as possible if it's looking at manipulating people.
  - tristor 6mo ago
    > You kind of put yourself in a trap thinking AI will behave as smart as possible if it's looking at manipulating people. Not at all. You're missing my point. Intelligence (even super intelligence) is not enough, because we already have that and it doesn't really result in outsized impacts. Our social structures are designed so that power and wealth accrue to the top and incumbency advantage outplays almost everything else. The only way in which ASI creates any impacts is to accelerate what is already happening if it can be thoroughly reined in by those already in power, otherwise it really doesn't seem to me that it will do much. My AI doomer take is that we're going to (we already are, actually) shoot ourselves in the foot by making everything worse for no benefit by getting rid of actual human experts and replacing them with non-intelligent models, causing a major backslide in society-level capabilities across the board, because the people in power are too stupid to know the difference. I am personally witnessing this in real-time in multiple parts of the tech industry. You give /wayyyy/ too much credit to those in power, that they are "playing dumb". Not really, they are actually dumb, in some cases severely so. I am not saying this an external observer who is watching a sound-bite on television, I am saying this as someone who is regularly in the room with very senior people across industry and am utterly shocked at the complete lack of competence and understanding of the core technologies they're theoretically responsible for shepherding. It's not surprising at all to me that they believe claims about LLMs that are clearly false, because they lack the necessary technical literacy to evaluate those claims and the LLMs perfectly fit into their optimization around yes-men, so they're happy to believe them whether they're true or false, as they see themselves as insulated from any consequences. More than 300k people have been laid off across the tech industry in the last 18 months, most of them accompanying claims of "AI", when in actuality, no net positive impacts have been seen, either for the companies themselves or any of the people that remain. So, yeah, not really concerned with ASI / Terminator scenarios, we're going to fuck ourselves over long before we get there just out of Dunning-Kruger and general MBA stupidity.