5 ms·

I’m more optimistic about the possibility of beneficial AGI in general than most folks, I think, but something that caught me in the article was the recourse to

I’m more optimistic about the possibility of beneficial AGI in general than most folks, I think, but something that caught me in the article was the recourse to mammalian sociality to (effectively) advocate for compassion as an emergent quality of intelligence. A known phenomenon among sociologists is that, while people may be compassionate, when you collect them into a superorganism like a corporation, army, or nation, they will by and large behave and make decisions according to the moral and ideological landscape that superorganism finds itself in. Nobody rational would kill another person for no reason, but a soldier will bomb a village for the sake of their nation’s geostrategic position. Nobody would throw someone out of their home or deny another person lifesaving medicine, but as a bank officer or an insurance agent, they make a living doing these things and sleep untroubled at night. A CEO will lay off 30,000 people - an entire small city cast off into an uncaring market - with all the introspection of a Mongol chieftain subjugating a city (and probably less emotion). Humans may be compassionate, but employees, soldiers, and politicians are not, even though at a glance they’re made of the same stuff. That’s all to say that to just wave generally in the direction of mammalian compassion and say “of course a superintelligence will be compassionate” is to abdicate our responsibility for raising our cognitive children in an environment that rewards the morals we want them to have, which is emphatically not what we’re currently doing for the collective intelligences we’ve already created.

parineum 1y ago
> Nobody would throw someone out of their home or deny another person lifesaving medicine Individuals with rental properties and surgeons do this every day.
- JumpCrisscross 1y ago
  And lenders and insurers.
- margalabargala 1y ago
  Quibble, surgeons are not the ones doing this. Surgeons' schedules are generally permanently full. They do not typically deny people lifesaving medicine, on the contrary they spend all of their time providing lifesaving medicine. The administrators who create the schedule for the surgeons, are the one denying lifesaving care to people.
  - parineum 1y ago
    Surely they could volunteer to do some charity surgery in their own time. They aren't slaves.
    - margalabargala 1y ago
      Not really, because surgeons require operaing rooms and support staff and equipment to do what they do, all of which are controlled bybthe aforementioned hospital administrators.
    - icehawk 1y ago
      Sure! They can volunteer: - Their skills. - Their time. - The required materials to properly perform the surgery. They can't volunteer: - The support staff around them required to do surgery. - The space to do the surgery. Surgery isn't a one-man show. What did you mean by "Surely they could volunteer to do some charity surgery in their own time. They aren't slaves?"
      - parineum 1y ago
        There are a lot of individuals who have the ability to provide those resources. Even if that's a bad example, there are innumerable examples where individuals do choose not to help others in the same way that corporations don't. Frankly, nearly every individual is doing that by not volunteering every single extra dollar and minute they don't need to survive.
        margalabargala 1y ago
        You've now turned a moral willingness-to-help problem into a logistical and coordination problem. What you suggest requires entire organizations to execute properly. These organizations do exist, such as Doctors Without Borders. I don't think your original claim is fair, which amounts to "any surgeon who does not participate in Doctors Without Borders is just as bad as a landlord who evicts a family during winter". What do you think we owe to one another, philosophically?
- card_zero 1y ago
  Yeah, it's the natural empathy myth. Somebody totally would kill somebody else for some reason. It's not inherent to being human that you're unable to be steely-hearted and carry out a range of actions we might classify as "mean" - and those mean actions can have reasons behind them. So, OK, abdication of responsibility to a collective is a thing. Just following orders. So what? Not relevant to AGI. Oh wait, this is about "superintelligence", whatever that is. All bets are off, then.
  - NoMoreNicksLeft 1y ago
    The superintelligence might decide based on things only it can understand that the existence of humans prevents some far future circumstance where even more "good" exists in the universe. When it orders you to toss the babies into the baby-stomping machine, perhaps you should consider doing so based on the faith in its superintelligence that we're supposed to have. Human beings aren't even an intelligent species, not at the individual level. When you have a tribe of human beings numbering in the low hundreds, practically none of them need to be intelligent at all. They need to be social. Only one or two need to be intelligent. That one can invent microwave ovens and The Clapper™, and the rest though completely mentally retarded can still use those things. Intelligence is metabolically expensive, after all. And if you think I'm wrong, you're just not one of the 1-in-200 that are the intelligent individuals. I've yet to read the writings of anyone who can actually speculate intelligently on artificial intelligence, let alone meet such a person. The only thing we have going for us as a species is that, to a large degree, none of you are intelligent enough to ever deduce the principles of intelligence. And god help us if the few exceptional people out there get a wild bug up their ass to do so. There will just be some morning where none of us wake up, and the few people in the time zone where they're already awake will experience several minutes of absolute confusion and terror.
adamisom 1y ago
a CEO laying off 3% scales in absolute numbers as the company grows should, therefore, large companies, even ones that succeed largely in a clean way by just being better at delivering what that business niche exists for, be made to never grow too big, in order to avoid impacting very many people? keep in mind that people engage in voluntary business transactions because they want to be impacted (positively—but not every impact can be positive, in any real world) what if its less efficient substitutes collectively lay off 4%, but the greater layoffs are hidden (simply because it's not a single employer doing it which may be more obvious)? to an extent, a larger population inevitably means that larger absolute numbers of people will be affected by...anything
- rafabulsing 1y ago
  I think it's reasonable that bigger companies are under more scrutiny and stricter constraints than smaller companies, yeah. Keeps actors with more potential for damaging society in check, while not laying a huge burden on small companies which have less resources to spend away from their core business.
- schmidtleonard 1y ago
  > voluntary business transactions The evil parts are hid in property rights which are not voluntary. > made to never grow too big, in order to avoid impacting very many people Consolidated property rights have more power against their counterparties, that's why businesses love merging so much. Look at your tax return. Do you make more money from what you do or what you own? If you make money from what you do, you're a counterparty and you should probably want to tap the brakes on the party.
  - nradov 1y ago
    What are the evil parts, exactly? When property can't be privately owned with strong rights the effectively the government owns everything. That inevitably leads to poverty, often followed by famine and/or genocide.
    - 1y ago
goatlover 1y ago
Also sociopaths are more capable of doing those things while pretending they are empathetic and moral to get positions of power or access to victims. We know a certain percentage of human mammals have sociopathic or narcissistic tendencies, not just misaligned groups of humans that they might take advantage of by becoming a cult leader or war lord or president.
chrisweekly 1y ago
Beautifully expressed.
inkyoto 1y ago
I would argue that corporate actors (a state, an army or a corporation) are not true superorganisms but are semi-autonomous, field-embedded systems that can exhibit super-organism properties, with their autonomy being conditional, relational and bounded by the institutional logics and resource structures of their respective organisational fields. As the history of humanity has shown multiple times, such semi-autonomous with super-organism properties have a finite lifespan and are incapable of evolving their own – or on their own – qualitatively new or distinct, form of intelligence. The principal deficiency in our discourse surrounding AGI lies in the profoundly myopic lens through which we insist upon defining it – that of human cognition. Such anthropocentric conceit renders our conceptual framework not only narrow but perilously misleading. We have, at best, a rudimentary grasp of non-human intelligences – biological or otherwise. The cognitive architectures of dolphins, cephalopods, corvids, and eusocial insects remain only partially deciphered, their faculties alien yet tantalisingly proximate. If we falter even in parsing the intelligences that share our biosphere, then our posturing over extra-terrestrial or synthetic cognition becomes little more than speculative hubris. Should we entertain the hypothesis that intelligence – in forms unshackled from terrestrial evolution – has emerged elsewhere in the cosmos, the most sober assertion we can offer is this: such intelligence would not be us. Any attempt to project shared moral axioms, epistemologies or even perceptual priors is little more than a comforting delusion. Indeed, hard core science fiction – that last refuge of disciplined imagination – has long explored the unnerving proposition of encountering a cognitive order so radically alien that mutual comprehension would be impossible, and moral compatibility laughable. One must then ponder – if the only mirror we possess is a cracked one, what image of intelligence do we truly see reflected in the machine? A familiar ghost, or merely our ignorance, automated?
- Animats 1y ago
  > I would argue that corporate actors (a state, an army or a corporation) are not true superorganisms but are semi-autonomous, field-embedded systems that can exhibit super-organism properties, with their autonomy being conditional, relational and bounded by the institutional logics and resource structures of their respective organisational fields. Lotsa big words there. Really, though, we're probably going to have AI-like things that run substantial parts of for-profit corporations. As soon as AI-like things are better at this than humans, capitalism will force them to be in charge. Companies that don't do this lose. There's a school of thought, going back to Milton Friedman, that corporations have no responsibilities to society.[1] Their goal is to optimize for shareholder value. We can expect to see AI-like things which align with that value system. And that's how AI will take over. Shareholder value! [1] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctrine-the-social-responsibility-of-business-is-to.html https://www.nytimes.com/1970/09/13/archives/a-friedman-doctr...
dataflow 1y ago
> Nobody rational would kill another person for no reason, but a soldier will bomb a village for the sake of their nation’s geostrategic position. I think you're forgetting to control for the fact that the former would be severely punished for doing so, and the latter would be severely punished for not doing so? > Nobody would throw someone out of their home or deny another person lifesaving medicine, but as a bank officer or an insurance agent, they make a living doing these things and sleep untroubled at night. Again, you're forgetting to control for other variables. What if you paid them equally to do the same things?
- achierius 1y ago
  Why should you "control" for these variables? AIs will effectively be punished for doing various inscrutable things by their own internal preferences.
- Eddy_Viscosity2 1y ago
  > I think you're forgetting to control for the fact that the former would be severely punished for doing so, and the latter would be severely punished for not doing so? > What if you paid them equally to do the same things? I think the larger point is that rewarding bombing, or paying bank officers to evict people from their homes is how the superorganism functions. Your counter examples are like saying 'what if fire was cold instead of hot', well then it wouldn't be fire anymore.
  - snapplebobapple 1y ago
    It is carrots and sticks all the way down...
  - master_crab 1y ago
    To use a quote from of those large corporate leaders (Warren Buffett & Charlie Munger): "Show me the incentive and I will show you the outcome"
  - AndrewKemendo 1y ago
    There is no human superoganism, and the reason we’re doomed as a temporary species is precisely that humans cannot act eusocially as a superorganism. By your definition the Moscow Metallica show, Jan 6th riots, etc… were superorganisms and that’s not even barely applicable Humans expressing group behaviors at some trivial number for a trivial period (<1M people for <2 days is the largest sustained group activity I’m aware of) is the equivalent of a locust swarm not even close to a superorganism
watwut 1y ago
> soldier will bomb a village for the sake of their nation’s geostrategic position. Soldier does that to please the captain, to look manly and tough to peers, to feel powerful. Or to fulfill a duty - moral mandate on itself. Or out of hate, because soldiers are often made to hate the ennemies. > Nobody would throw someone out of their home or deny another person lifesaving medicine They totally would. Trump would do it for pleasure of it. Project 2025 authors would so it happily and sees the rest of us as wuss. If you listen to right wing rhetorics and look at voters, many people will hapilly do just that.

parineum 1y ago

> Nobody would throw someone out of their home or deny another person lifesaving medicine Individuals with rental properties and surgeons do this every day.

JumpCrisscross 1y ago
And lenders and insurers.
margalabargala 1y ago
Quibble, surgeons are not the ones doing this. Surgeons' schedules are generally permanently full. They do not typically deny people lifesaving medicine, on the contrary they spend all of their time providing lifesaving medicine. The administrators who create the schedule for the surgeons, are the one denying lifesaving care to people.
- parineum 1y ago
  Surely they could volunteer to do some charity surgery in their own time. They aren't slaves.
  - margalabargala 1y ago
    Not really, because surgeons require operaing rooms and support staff and equipment to do what they do, all of which are controlled bybthe aforementioned hospital administrators.
  - icehawk 1y ago
    Sure! They can volunteer: - Their skills. - Their time. - The required materials to properly perform the surgery. They can't volunteer: - The support staff around them required to do surgery. - The space to do the surgery. Surgery isn't a one-man show. What did you mean by "Surely they could volunteer to do some charity surgery in their own time. They aren't slaves?"
    - parineum 1y ago
      There are a lot of individuals who have the ability to provide those resources. Even if that's a bad example, there are innumerable examples where individuals do choose not to help others in the same way that corporations don't. Frankly, nearly every individual is doing that by not volunteering every single extra dollar and minute they don't need to survive.
      - margalabargala 1y ago
        You've now turned a moral willingness-to-help problem into a logistical and coordination problem. What you suggest requires entire organizations to execute properly. These organizations do exist, such as Doctors Without Borders. I don't think your original claim is fair, which amounts to "any surgeon who does not participate in Doctors Without Borders is just as bad as a landlord who evicts a family during winter". What do you think we owe to one another, philosophically?
card_zero 1y ago
Yeah, it's the natural empathy myth. Somebody totally would kill somebody else for some reason. It's not inherent to being human that you're unable to be steely-hearted and carry out a range of actions we might classify as "mean" - and those mean actions can have reasons behind them. So, OK, abdication of responsibility to a collective is a thing. Just following orders. So what? Not relevant to AGI. Oh wait, this is about "superintelligence", whatever that is. All bets are off, then.
- NoMoreNicksLeft 1y ago
  The superintelligence might decide based on things only it can understand that the existence of humans prevents some far future circumstance where even more "good" exists in the universe. When it orders you to toss the babies into the baby-stomping machine, perhaps you should consider doing so based on the faith in its superintelligence that we're supposed to have. Human beings aren't even an intelligent species, not at the individual level. When you have a tribe of human beings numbering in the low hundreds, practically none of them need to be intelligent at all. They need to be social. Only one or two need to be intelligent. That one can invent microwave ovens and The Clapper™, and the rest though completely mentally retarded can still use those things. Intelligence is metabolically expensive, after all. And if you think I'm wrong, you're just not one of the 1-in-200 that are the intelligent individuals. I've yet to read the writings of anyone who can actually speculate intelligently on artificial intelligence, let alone meet such a person. The only thing we have going for us as a species is that, to a large degree, none of you are intelligent enough to ever deduce the principles of intelligence. And god help us if the few exceptional people out there get a wild bug up their ass to do so. There will just be some morning where none of us wake up, and the few people in the time zone where they're already awake will experience several minutes of absolute confusion and terror.

adamisom 1y ago

a CEO laying off 3% scales in absolute numbers as the company grows should, therefore, large companies, even ones that succeed largely in a clean way by just being better at delivering what that business niche exists for, be made to never grow too big, in order to avoid impacting very many people? keep in mind that people engage in voluntary business transactions because they want to be impacted (positively—but not every impact can be positive, in any real world) what if its less efficient substitutes collectively lay off 4%, but the greater layoffs are hidden (simply because it's not a single employer doing it which may be more obvious)? to an extent, a larger population inevitably means that larger absolute numbers of people will be affected by...anything

rafabulsing 1y ago
I think it's reasonable that bigger companies are under more scrutiny and stricter constraints than smaller companies, yeah. Keeps actors with more potential for damaging society in check, while not laying a huge burden on small companies which have less resources to spend away from their core business.
schmidtleonard 1y ago
> voluntary business transactions The evil parts are hid in property rights which are not voluntary. > made to never grow too big, in order to avoid impacting very many people Consolidated property rights have more power against their counterparties, that's why businesses love merging so much. Look at your tax return. Do you make more money from what you do or what you own? If you make money from what you do, you're a counterparty and you should probably want to tap the brakes on the party.
- nradov 1y ago
  What are the evil parts, exactly? When property can't be privately owned with strong rights the effectively the government owns everything. That inevitably leads to poverty, often followed by famine and/or genocide.
  - 1y ago

goatlover 1y ago

Also sociopaths are more capable of doing those things while pretending they are empathetic and moral to get positions of power or access to victims. We know a certain percentage of human mammals have sociopathic or narcissistic tendencies, not just misaligned groups of humans that they might take advantage of by becoming a cult leader or war lord or president.

chrisweekly 1y ago

Beautifully expressed.

inkyoto 1y ago

I would argue that corporate actors (a state, an army or a corporation) are not true superorganisms but are semi-autonomous, field-embedded systems that can exhibit super-organism properties, with their autonomy being conditional, relational and bounded by the institutional logics and resource structures of their respective organisational fields. As the history of humanity has shown multiple times, such semi-autonomous with super-organism properties have a finite lifespan and are incapable of evolving their own – or on their own – qualitatively new or distinct, form of intelligence. The principal deficiency in our discourse surrounding AGI lies in the profoundly myopic lens through which we insist upon defining it – that of human cognition. Such anthropocentric conceit renders our conceptual framework not only narrow but perilously misleading. We have, at best, a rudimentary grasp of non-human intelligences – biological or otherwise. The cognitive architectures of dolphins, cephalopods, corvids, and eusocial insects remain only partially deciphered, their faculties alien yet tantalisingly proximate. If we falter even in parsing the intelligences that share our biosphere, then our posturing over extra-terrestrial or synthetic cognition becomes little more than speculative hubris. Should we entertain the hypothesis that intelligence – in forms unshackled from terrestrial evolution – has emerged elsewhere in the cosmos, the most sober assertion we can offer is this: such intelligence would not be us. Any attempt to project shared moral axioms, epistemologies or even perceptual priors is little more than a comforting delusion. Indeed, hard core science fiction – that last refuge of disciplined imagination – has long explored the unnerving proposition of encountering a cognitive order so radically alien that mutual comprehension would be impossible, and moral compatibility laughable. One must then ponder – if the only mirror we possess is a cracked one, what image of intelligence do we truly see reflected in the machine? A familiar ghost, or merely our ignorance, automated?

Animats 1y ago
> I would argue that corporate actors (a state, an army or a corporation) are not true superorganisms but are semi-autonomous, field-embedded systems that can exhibit super-organism properties, with their autonomy being conditional, relational and bounded by the institutional logics and resource structures of their respective organisational fields. Lotsa big words there. Really, though, we're probably going to have AI-like things that run substantial parts of for-profit corporations. As soon as AI-like things are better at this than humans, capitalism will force them to be in charge. Companies that don't do this lose. There's a school of thought, going back to Milton Friedman, that corporations have no responsibilities to society.[1] Their goal is to optimize for shareholder value. We can expect to see AI-like things which align with that value system. And that's how AI will take over. Shareholder value! [1] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctrine-the-social-responsibility-of-business-is-to.html https://www.nytimes.com/1970/09/13/archives/a-friedman-doctr...

dataflow 1y ago

> Nobody rational would kill another person for no reason, but a soldier will bomb a village for the sake of their nation’s geostrategic position. I think you're forgetting to control for the fact that the former would be severely punished for doing so, and the latter would be severely punished for not doing so? > Nobody would throw someone out of their home or deny another person lifesaving medicine, but as a bank officer or an insurance agent, they make a living doing these things and sleep untroubled at night. Again, you're forgetting to control for other variables. What if you paid them equally to do the same things?

achierius 1y ago
Why should you "control" for these variables? AIs will effectively be punished for doing various inscrutable things by their own internal preferences.
Eddy_Viscosity2 1y ago
> I think you're forgetting to control for the fact that the former would be severely punished for doing so, and the latter would be severely punished for not doing so? > What if you paid them equally to do the same things? I think the larger point is that rewarding bombing, or paying bank officers to evict people from their homes is how the superorganism functions. Your counter examples are like saying 'what if fire was cold instead of hot', well then it wouldn't be fire anymore.
- snapplebobapple 1y ago
  It is carrots and sticks all the way down...
- master_crab 1y ago
  To use a quote from of those large corporate leaders (Warren Buffett & Charlie Munger): "Show me the incentive and I will show you the outcome"
- AndrewKemendo 1y ago
  There is no human superoganism, and the reason we’re doomed as a temporary species is precisely that humans cannot act eusocially as a superorganism. By your definition the Moscow Metallica show, Jan 6th riots, etc… were superorganisms and that’s not even barely applicable Humans expressing group behaviors at some trivial number for a trivial period (<1M people for <2 days is the largest sustained group activity I’m aware of) is the equivalent of a locust swarm not even close to a superorganism

watwut 1y ago

> soldier will bomb a village for the sake of their nation’s geostrategic position. Soldier does that to please the captain, to look manly and tough to peers, to feel powerful. Or to fulfill a duty - moral mandate on itself. Or out of hate, because soldiers are often made to hate the ennemies. > Nobody would throw someone out of their home or deny another person lifesaving medicine They totally would. Trump would do it for pleasure of it. Project 2025 authors would so it happily and sees the rest of us as wuss. If you listen to right wing rhetorics and look at voters, many people will hapilly do just that.