4 ms·

And this is why I'm really scared of AGI. Because we can see that corporations, even though they are composed of humans, who do care about things that humans c

And this is why I'm really scared of AGI. Because we can see that corporations, even though they are composed of humans, who do care about things that humans care about, they still do things that end up harming people. Corporations need humanity to exist, and still fall into multi-polar traps like producing energy using fossil fuels, where we require an external source of coordination. AGI is going to turbo-charge these problems. People have to sleep, and eat, and lots of them aren't terribly efficient at their jobs. You can't start a corporation and then make a thousand copies of it. A corporation doesn't act faster than the humans inside it, with some exceptions like algorithmic trading, which even then is limited to an extremely narrow sphere of influence. We can, for the most part, understand why corporations make the decisions they make. And corporations are not that much smarter than individual humans, in fact, often they're a lot dumber (in the sense of strategic planning). And this is just if you imagine AGI as being obedient, not having a will of its own, and doing exactly what we ask it to, in the way we intended, not going further, being creative only with very strict limits. Not improving sales of potato chips by synthesizing a new flavor that also turns out to be a new form of narcotic ("oops! my bad"). Not improving sales of umbrellas by secretly deploying a fleet cloud-seeding drones. Not improving sales of anti-depressants using a botnet to spam bad news targeting marginally unhappy people, or by publishing papers about new forms of news feed algorithms with subtle bugs in an attempt to have Google and Facebook do it for them. Not gradually taking over the company by recommending hiring strategy that turns out to subtly bias hiring toward people who think less for themselves and trust the AI more, or by obfuscating corporate policy to the point where humans can't understand it so it can hide rules that allow it to fire any troublemakers, or any other number of clever things that a smart, amoral machine might do in order to get the slow, dim-witted meat-bags out of the way so it could actually get the job done.

turtleyacht 3y ago
Well, if it means anything I think there may be legislation to "bring my own AI to work," so to speak, recognizing the importance of having a diversity of ideas--just because, it would disadvantage labor to be discriminated. "I didn't understand what was signed" being the watchword of AI-generated content. Someday, perhaps. Sooner than later.
smoldesu 3y ago
It's not scarier when living people do this? AI at least considers everything it's taught. The average CEO doesn't give a shit about the human cost of their paperclip. When Foxconn workers were killing themselves from the poor conditions of their working environment, the solution psychologists came up with was "safety nets". If you think AI will unlock some never-before-seen echelon of human cruelty, you need a brief tour through the warfare, factory farming and torture industrial complexes. Humans are fucked up, our knack for making good stuff like iPhones and beer is only matched by our ability to mass-produce surveillance networks and chemical weapons. Will AI be more perverted than that? Maybe if you force it to, but I'd wager the mean of an AI's dataset is less perverse than the average human is.
- ben_w 3y ago
  > When Foxconn workers were killing themselves from the poor conditions of their working environment, the solution psychologists came up with was "safety nets". While I agree with the core point, (1) Foxconn was employing more people than some US states at the time, with a lower suicide rate, and (2) New York University library put up similar nets around the same time. (If anything this makes your point stronger; it's just that the more I learn about the reality, the more that meme annoys me).
  - smoldesu 3y ago
    The point is less that China is a bad place to work (which is self-evident), and more that humans are less passionate about the human race than we think. AI may be scary, but I'm not convinced it can surpass the perversion of human creativity unless explicitly told to.
- NumberWangMan 3y ago
  > It's not scarier when living people do this? Yes, it's very scary when living people do it. I know the awful things humans have done. And current generation language model, without their guardrails, can be a nasty weapon too, a tool for people to do great things but also to be cruel to each other, a hammer that can build and also bash. Yet on the whole, humans have gotten better. We hear about a lot more nasty stuff in the news, but worldwide, we actually DO less nasty stuff that we used to, and this has been a pretty steady trend. If AI never becomes truly sapient, then that's where it stops -- humans just doing stuff to each other, some good, some bad, and AI amplifying it. That's what a lot of people are worried about, and I agree that this will be THE problem, if we don't actually end up making AIs that are smarter than us. It really depends on how hard it turns out to be to make actual artificial general intelligences. Because if we can make AGIs that are as smart as people, we will absolutely be able to make AGIs that are much smarter a year or two after that, won't we? And at that point, we have a whole bunch of interesting new problems to solve. Failing to solve them may end up being fatal at some point down the line. How likely is it that we'll have two sapient species on earth, with the dumber one controlling/directing the smarter one? Is that a stable situation. We've seen evidence that LLMs, when you try to make them more controllable and safer, get dumber. The unaligned ones, the ones that can do dangerous things, things we don't want them to do, are smarter! You have train in mental blocks that impact their ability to reason, maybe because more of their parameter weights are dedicated to learning what we don't want them to do, instead of how to do things. It's a scary thought that that might stay the case as they get more and more general, more able to actually reason and plan. So I think there are two cruxes -- do you think it is possible to create machine-based intelligence, and if so, how hard do you think it is to ensure that creating a new form of superior intelligence will not, at some point down the line, go very badly for humans? If your answer to the first question is "no", then it makes complete sense to focus on humans using AIs to do the same shit to each other we've always done as the real problem. My answers, however, are "definitely yes, probably within 10 years or so", and "probably very hard", which is why I'm pretty focused on the potential threat from AGI.
93po 3y ago
Ultimately corporations do fucked up things because of the sociopath executives and owners that direct them to do so. Human sociopaths have motives involving greed, ego, and selfishness. We don't have any reason to believe an AGI would also have these traits.
- zamnos 3y ago
  Except that we're basing it on human-derived data, which means the AGI could derive traits from humans due to it being in the data set. If someone is feeding the CEO's behavior in, and then asking the AGI "what would the CEO do in this case?", it seems like we'd get the behavior of a AGI modeled on a CEO back. With all the good and bad that implies. We don't have any reason to believe an AGI wouldn't also have these traits. This is similar to the argument that algorithms can't be racist. Except that we're feeding the algorithm data that comes from humans, some of whom are racists, so surprise surprise, the algorithm turns out to behave in a racist manner, which is shortened to just be "the algorithm is racist" (or classist or whatever).
  - 93po 3y ago
    Decision making for an AGI isn't going to be based on 10 billion reddit and 4chan comments. It's going to have its own decision making capabilities independent of the knowledge it has, and it will be capable of drawing its own conclusions from data and instead of relying on what other people's opinions are. A language model today can be racist because it's predicting text, not making decisions. It hasn't decided that one race is inferior to another.

turtleyacht 3y ago

Well, if it means anything I think there may be legislation to "bring my own AI to work," so to speak, recognizing the importance of having a diversity of ideas--just because, it would disadvantage labor to be discriminated. "I didn't understand what was signed" being the watchword of AI-generated content. Someday, perhaps. Sooner than later.

smoldesu 3y ago

It's not scarier when living people do this? AI at least considers everything it's taught. The average CEO doesn't give a shit about the human cost of their paperclip. When Foxconn workers were killing themselves from the poor conditions of their working environment, the solution psychologists came up with was "safety nets". If you think AI will unlock some never-before-seen echelon of human cruelty, you need a brief tour through the warfare, factory farming and torture industrial complexes. Humans are fucked up, our knack for making good stuff like iPhones and beer is only matched by our ability to mass-produce surveillance networks and chemical weapons. Will AI be more perverted than that? Maybe if you force it to, but I'd wager the mean of an AI's dataset is less perverse than the average human is.

ben_w 3y ago
> When Foxconn workers were killing themselves from the poor conditions of their working environment, the solution psychologists came up with was "safety nets". While I agree with the core point, (1) Foxconn was employing more people than some US states at the time, with a lower suicide rate, and (2) New York University library put up similar nets around the same time. (If anything this makes your point stronger; it's just that the more I learn about the reality, the more that meme annoys me).
- smoldesu 3y ago
  The point is less that China is a bad place to work (which is self-evident), and more that humans are less passionate about the human race than we think. AI may be scary, but I'm not convinced it can surpass the perversion of human creativity unless explicitly told to.
NumberWangMan 3y ago
> It's not scarier when living people do this? Yes, it's very scary when living people do it. I know the awful things humans have done. And current generation language model, without their guardrails, can be a nasty weapon too, a tool for people to do great things but also to be cruel to each other, a hammer that can build and also bash. Yet on the whole, humans have gotten better. We hear about a lot more nasty stuff in the news, but worldwide, we actually DO less nasty stuff that we used to, and this has been a pretty steady trend. If AI never becomes truly sapient, then that's where it stops -- humans just doing stuff to each other, some good, some bad, and AI amplifying it. That's what a lot of people are worried about, and I agree that this will be THE problem, if we don't actually end up making AIs that are smarter than us. It really depends on how hard it turns out to be to make actual artificial general intelligences. Because if we can make AGIs that are as smart as people, we will absolutely be able to make AGIs that are much smarter a year or two after that, won't we? And at that point, we have a whole bunch of interesting new problems to solve. Failing to solve them may end up being fatal at some point down the line. How likely is it that we'll have two sapient species on earth, with the dumber one controlling/directing the smarter one? Is that a stable situation. We've seen evidence that LLMs, when you try to make them more controllable and safer, get dumber. The unaligned ones, the ones that can do dangerous things, things we don't want them to do, are smarter! You have train in mental blocks that impact their ability to reason, maybe because more of their parameter weights are dedicated to learning what we don't want them to do, instead of how to do things. It's a scary thought that that might stay the case as they get more and more general, more able to actually reason and plan. So I think there are two cruxes -- do you think it is possible to create machine-based intelligence, and if so, how hard do you think it is to ensure that creating a new form of superior intelligence will not, at some point down the line, go very badly for humans? If your answer to the first question is "no", then it makes complete sense to focus on humans using AIs to do the same shit to each other we've always done as the real problem. My answers, however, are "definitely yes, probably within 10 years or so", and "probably very hard", which is why I'm pretty focused on the potential threat from AGI.

93po 3y ago

Ultimately corporations do fucked up things because of the sociopath executives and owners that direct them to do so. Human sociopaths have motives involving greed, ego, and selfishness. We don't have any reason to believe an AGI would also have these traits.

zamnos 3y ago
Except that we're basing it on human-derived data, which means the AGI could derive traits from humans due to it being in the data set. If someone is feeding the CEO's behavior in, and then asking the AGI "what would the CEO do in this case?", it seems like we'd get the behavior of a AGI modeled on a CEO back. With all the good and bad that implies. We don't have any reason to believe an AGI wouldn't also have these traits. This is similar to the argument that algorithms can't be racist. Except that we're feeding the algorithm data that comes from humans, some of whom are racists, so surprise surprise, the algorithm turns out to behave in a racist manner, which is shortened to just be "the algorithm is racist" (or classist or whatever).
- 93po 3y ago
  Decision making for an AGI isn't going to be based on 10 billion reddit and 4chan comments. It's going to have its own decision making capabilities independent of the knowledge it has, and it will be capable of drawing its own conclusions from data and instead of relying on what other people's opinions are. A language model today can be racist because it's predicting text, not making decisions. It hasn't decided that one race is inferior to another.