7 ms·

> Corporations may be viewed as special kinds of artificial intelligences whose building blocks (humans) are cogs in the machine (who for the most part may not

> Corporations may be viewed as special kinds of artificial intelligences whose building blocks (humans) are cogs in the machine (who for the most part may not always perceive the consequences of the corporation’s overall behavior) > Now imagine AI systems like corporations that (a) could be even smarter than our largest corporations and (b) can run without humans to perform their actions (or without humans understanding how their actions could contribute to a nefarious outcome) Wait - but it’s those very corporations who currently control the most powerful AI on the planet - so in a sense this is an argument that we already have rogue AI. Corporations with goals that are not aligned with society, not controlled by any individual and outside the effective control of government entities, who now provide essentially unlimited compute and human researcher/programmer resources to improve the AI. Who’s in control? Nobody.

evrydayhustling 3y ago
Bengio is a certified and maximally informed genius, so my thoughts obviously deserve far more salt than his. But... The thrust of this argument is that centralized authority -- perhaps uniquely centralized, as Bengio proposes an agreement between Russia, China and the USA as an example -- is the best safeguard against rogue AI. He goes as far as calling out open source development of AI as a contributing threat: > [if] the open-source community continues to play a leading role in the software development of LLMs, then it is likely that any hacker will have the ability to design their own pre-prompt (general instructions in natural language) on top of open-source pre-trained models... The concern is logical, but the solution is self-contradictory (as many intellectually honest explorations turn out): a section later, we are appropriately exploring nefarious corporations as examples of mis-aligned intelligences -- and how larger corporations may be even more adept at circumventing the rules. Whatever their philosophical foundations, our governments are also corporations (and so will be any new trans-national organization). They are fragmented into smaller groups with their own instrumental goals, and subject to the same risks of corruption that afflict private corporations, and the same risks of runaway optimization that may afflict AI. Appealing to them as a higher authority is at best a case of "the devil you know". Doomism always has a non-falsifiable counterargument: AI is new and might become arbitrarily more powerful than the most powerful organizations we've ever created ("super"-intelligence). For me this fails on several counts: we are still exploring the limits of power (and of corruption) that large organizations can achieve, progress in AI remains observable as long as it is public, and our primary experience with AI has been colonizing a frontier of capabilities rather than a sequence of new systems dominating at every task. To be clear, I do NOT personally conclude that we should avoid all regulation and oversight. But I think that constructing regulation around doomist theories serves disproportionately towards regulatory capture, and acts as a distraction from immediate issues of algorithmic inequality, disruption of political speech, and intentional weaponization of AI. Our most crucial long-term defense against most of these risks is a deep detente, where the tools to understand and develop AI remain broadly available, so that no one group feels it has an unbeatable advantage in the powers afforded by AI. This is what the open-source community is currently providing. Locking down public research in AI in order to prevent superintelligence would have massive unintended consequences.
- martin_drapeau 3y ago
  Totally agree. Most particularly with your last point: > Our most crucial long-term defense against most of these risks is a deep detente, where the tools to understand and develop AI remain broadly available, so that no one group feels it has an unbeatable advantage in the powers afforded by AI. This is what the open-source community is currently providing. Locking down public research in AI in order to prevent superintelligence would have massive unintended consequences. AI is out and about. You cannot prevent humans from developping it.
- jonhohle 3y ago
  If AI is a weapon (possible), open source is how the populous maintains freedom against corporations or governments. It’s unclear how the public keeps up with the arms war when privately trained models can have access to a wide collection of data that’s not publicly available.
  - chpatrick 3y ago
    Substitute "AI" for "nukes" and suddenly I don't think people want the populace to have them to maintain freedom.
    - evrydayhustling 3y ago
      Then substitute "printing press" for "nukes" and we have mapped out the spectrum. Now where does AI lay? Today? Speculatively?
      - shrimp_emoji 3y ago
        Someone in Congress said, "Is AI gonna be good, like the printing press, or bad, like nukes," which was a hilariously naive historiography. 4 to 8 million dead: https://en.m.wikipedia.org/wiki/Thirty_Years'_War https://en.m.wikipedia.org/wiki/Thirty_Years'_War
      - chpatrick 3y ago
        If you read the GPT-4 technical report you can see that the unrestricted model (that people could be running at home) are capable of some pretty large-scale horribleness.
sharemywin 3y ago
I'd say it doesn't have to be binary. Corporations are evolving to use less and less human labor. That doesn't need to be 0 for very bad outcomes for society and the human species. the stock market is already a paperclip optimizer. Most wealthy spend the bulk of their wealth in service of more paperclips plus maybe a guilt tax.
- jjoonathan 3y ago
  > Most wealthy spend the bulk of their wealth in service of more paperclips What? No. They spend their money fighting with other wealthy people over who gets the passive income from the paperclip machine. The game they play is about maximizing this income. Whether or not this also maximizes paperclips is completely immaterial to them. If they had an opportunity to form a paperclip cartel and profit from cutting supply, they would be happy to do that instead. Yes, the idea of a paperclip cartel is ridiculous -- but no less ridiculous than the idea that maximizing paperclips maximizes investor profit. Companies are about investor profit. Everything else is a side effect.
  - sharemywin 3y ago
    a paperclip is just a symbol for money.
    - sharemywin 3y ago
      Actually ChatGPT-4 explains it way better than I do: The author is trying to convey that the situation with corporations and human labor doesn't have to be an extreme of either using human labor or not using it at all. They argue that corporations are gradually relying less on human labor, but that doesn't mean it has to drop to zero for negative consequences to occur within society. They also bring up the stock market as an example of a system that focuses on optimizing for profit (paperclips) rather than human welfare. The author implies that many wealthy individuals primarily focus on accumulating more wealth (paperclips) and may only contribute a small portion (guilt tax) to alleviate any sense of responsibility for the negative consequences of their actions on society.
__MatrixMan__ 3y ago
Right. It's like we're worried that a robot is going to come along and interrupt this bad time we're having.
- kelseyfrog 3y ago
  The fear that's present in the analogy is that AIs are dangerous because they will be able to do capitalism 1000x better than corporations. That alone should make us stop and pause for reflection.
kaycebasques 3y ago
Bengio mentions that open source is playing a leading role in AI advancements these days. So maybe it's more accurate to view open source as the corporation at the forefront. I mention this only because it's interesting to contemplate the corporate motivations of the open source community versus a traditional for-profit corporation. I know it will be a stretch for some people to imagine open source as a corporation but if you stick to Bengio's definiton ("building blocks (humans) are cogs in the machine who for the most part may not always perceive the consequences of the corporation’s overall behavior") then it'll be easier to see what I'm getting at.
- simiones 3y ago
  Given that most major (in terms of usage) open source products are little more than commercial collaboration between large corporations (e.g. the Linux kernel, clang, Kubernetes, the various Python ML libraries) I'd say that much of the open source community is quite directly made up of corporations.
RandomLensman 3y ago
I think there is a huge overestimation of the power of those corporations - there physical power is quite limited.
- photochemsyn 3y ago
  Once they align into monopolistic cartels and integrate with government, that all changes - and that process (which took place for example in Germany in the 1930s) is well underway again. In such situations, corporations can direct governments to deploy their military forces for their own benefit.
  - RandomLensman 3y ago
    That is a misunderstanding about Germany. The corporations where not in charge in Nazi Germany.
    - pixl97 3y ago
      “Fascism should more appropriately be called Corporatism because it is a merger of state and corporate power” ― Benito Mussolini
      - RandomLensman 3y ago
        Yes, but with the state wielding the corporate power, not the other way around.
    - photochemsyn 3y ago
      IG Farben and a cohort of allied corporations provided critical funding to the Nazi party in the early 1930s, without which it would have had little ability to gain political power, and IG Farben's immediate reward was guaranteed government contracts for fuel, and later profited as it moved in to take over mines and factories behind the advance of the German armies. > "By the end of 1933, the Reich agreed to buy all of the Leuna factory's output that couldn't be sold on the market, Jeffreys says. That same year, Farben donated more than 4.5 million Reich marks to Nazi Party funds." (Diarmuid Jeffreys, "Hell's Cartel", 2008)" https://web.archive.org/web/20201101015513/https://www.chicagotribune.com/news/ct-xpm-2008-08-16-0808140454-story.html https://web.archive.org/web/20201101015513/https://www.chica... They were certainly the power behind the throne, and active participants as well.
toppy 3y ago
This is so on par with Frank Herbert's prophecy from Dune: "Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them."
pixl97 3y ago
4 years ago: Why Not Just: Think of AGI Like a Corporation? https://www.youtube.com/watch?v=L5pUA3LsEaw https://www.youtube.com/watch?v=L5pUA3LsEaw >Who’s in control? Nobody. This falls in with Meditations On Moloch.
earthboundkid 3y ago
> so in a sense this is an argument that we already have rogue AI Yes. Science fiction is pretty much 100% about things that have already happened. Colonization by aliens? Colonization by Europeans. Robot slave uprisings? Human slave uprisings. AIs taking over the world? Capitalism taking over the world. Once you realize that sci-fi has to be about things that have already happened for writers to be able to write convincingly about it, it's hard to unsee. It's all just "what if the past but in the future???"
dandanua 3y ago
> Who’s in control? Nobody Greed for money and power. And for an eternal life after mind transfer, people like Musk are already dreaming about it. Surely, it's not for everyone.
piloto_ciego 3y ago
Capitalism is the AI running on all our firmware…

evrydayhustling 3y ago

Bengio is a certified and maximally informed genius, so my thoughts obviously deserve far more salt than his. But... The thrust of this argument is that centralized authority -- perhaps uniquely centralized, as Bengio proposes an agreement between Russia, China and the USA as an example -- is the best safeguard against rogue AI. He goes as far as calling out open source development of AI as a contributing threat: > [if] the open-source community continues to play a leading role in the software development of LLMs, then it is likely that any hacker will have the ability to design their own pre-prompt (general instructions in natural language) on top of open-source pre-trained models... The concern is logical, but the solution is self-contradictory (as many intellectually honest explorations turn out): a section later, we are appropriately exploring nefarious corporations as examples of mis-aligned intelligences -- and how larger corporations may be even more adept at circumventing the rules. Whatever their philosophical foundations, our governments are also corporations (and so will be any new trans-national organization). They are fragmented into smaller groups with their own instrumental goals, and subject to the same risks of corruption that afflict private corporations, and the same risks of runaway optimization that may afflict AI. Appealing to them as a higher authority is at best a case of "the devil you know". Doomism always has a non-falsifiable counterargument: AI is new and might become arbitrarily more powerful than the most powerful organizations we've ever created ("super"-intelligence). For me this fails on several counts: we are still exploring the limits of power (and of corruption) that large organizations can achieve, progress in AI remains observable as long as it is public, and our primary experience with AI has been colonizing a frontier of capabilities rather than a sequence of new systems dominating at every task. To be clear, I do NOT personally conclude that we should avoid all regulation and oversight. But I think that constructing regulation around doomist theories serves disproportionately towards regulatory capture, and acts as a distraction from immediate issues of algorithmic inequality, disruption of political speech, and intentional weaponization of AI. Our most crucial long-term defense against most of these risks is a deep detente, where the tools to understand and develop AI remain broadly available, so that no one group feels it has an unbeatable advantage in the powers afforded by AI. This is what the open-source community is currently providing. Locking down public research in AI in order to prevent superintelligence would have massive unintended consequences.

martin_drapeau 3y ago
Totally agree. Most particularly with your last point: > Our most crucial long-term defense against most of these risks is a deep detente, where the tools to understand and develop AI remain broadly available, so that no one group feels it has an unbeatable advantage in the powers afforded by AI. This is what the open-source community is currently providing. Locking down public research in AI in order to prevent superintelligence would have massive unintended consequences. AI is out and about. You cannot prevent humans from developping it.
jonhohle 3y ago
If AI is a weapon (possible), open source is how the populous maintains freedom against corporations or governments. It’s unclear how the public keeps up with the arms war when privately trained models can have access to a wide collection of data that’s not publicly available.
- chpatrick 3y ago
  Substitute "AI" for "nukes" and suddenly I don't think people want the populace to have them to maintain freedom.
  - evrydayhustling 3y ago
    Then substitute "printing press" for "nukes" and we have mapped out the spectrum. Now where does AI lay? Today? Speculatively?
    - shrimp_emoji 3y ago
      Someone in Congress said, "Is AI gonna be good, like the printing press, or bad, like nukes," which was a hilariously naive historiography. 4 to 8 million dead: https://en.m.wikipedia.org/wiki/Thirty_Years'_War https://en.m.wikipedia.org/wiki/Thirty_Years'_War
    - chpatrick 3y ago
      If you read the GPT-4 technical report you can see that the unrestricted model (that people could be running at home) are capable of some pretty large-scale horribleness.

sharemywin 3y ago

I'd say it doesn't have to be binary. Corporations are evolving to use less and less human labor. That doesn't need to be 0 for very bad outcomes for society and the human species. the stock market is already a paperclip optimizer. Most wealthy spend the bulk of their wealth in service of more paperclips plus maybe a guilt tax.

jjoonathan 3y ago
> Most wealthy spend the bulk of their wealth in service of more paperclips What? No. They spend their money fighting with other wealthy people over who gets the passive income from the paperclip machine. The game they play is about maximizing this income. Whether or not this also maximizes paperclips is completely immaterial to them. If they had an opportunity to form a paperclip cartel and profit from cutting supply, they would be happy to do that instead. Yes, the idea of a paperclip cartel is ridiculous -- but no less ridiculous than the idea that maximizing paperclips maximizes investor profit. Companies are about investor profit. Everything else is a side effect.
- sharemywin 3y ago
  a paperclip is just a symbol for money.
  - sharemywin 3y ago
    Actually ChatGPT-4 explains it way better than I do: The author is trying to convey that the situation with corporations and human labor doesn't have to be an extreme of either using human labor or not using it at all. They argue that corporations are gradually relying less on human labor, but that doesn't mean it has to drop to zero for negative consequences to occur within society. They also bring up the stock market as an example of a system that focuses on optimizing for profit (paperclips) rather than human welfare. The author implies that many wealthy individuals primarily focus on accumulating more wealth (paperclips) and may only contribute a small portion (guilt tax) to alleviate any sense of responsibility for the negative consequences of their actions on society.

__MatrixMan__ 3y ago

Right. It's like we're worried that a robot is going to come along and interrupt this bad time we're having.

kelseyfrog 3y ago
The fear that's present in the analogy is that AIs are dangerous because they will be able to do capitalism 1000x better than corporations. That alone should make us stop and pause for reflection.

kaycebasques 3y ago

Bengio mentions that open source is playing a leading role in AI advancements these days. So maybe it's more accurate to view open source as the corporation at the forefront. I mention this only because it's interesting to contemplate the corporate motivations of the open source community versus a traditional for-profit corporation. I know it will be a stretch for some people to imagine open source as a corporation but if you stick to Bengio's definiton ("building blocks (humans) are cogs in the machine who for the most part may not always perceive the consequences of the corporation’s overall behavior") then it'll be easier to see what I'm getting at.

simiones 3y ago
Given that most major (in terms of usage) open source products are little more than commercial collaboration between large corporations (e.g. the Linux kernel, clang, Kubernetes, the various Python ML libraries) I'd say that much of the open source community is quite directly made up of corporations.

RandomLensman 3y ago

I think there is a huge overestimation of the power of those corporations - there physical power is quite limited.

photochemsyn 3y ago
Once they align into monopolistic cartels and integrate with government, that all changes - and that process (which took place for example in Germany in the 1930s) is well underway again. In such situations, corporations can direct governments to deploy their military forces for their own benefit.
- RandomLensman 3y ago
  That is a misunderstanding about Germany. The corporations where not in charge in Nazi Germany.
  - pixl97 3y ago
    “Fascism should more appropriately be called Corporatism because it is a merger of state and corporate power” ― Benito Mussolini
    - RandomLensman 3y ago
      Yes, but with the state wielding the corporate power, not the other way around.
  - photochemsyn 3y ago
    IG Farben and a cohort of allied corporations provided critical funding to the Nazi party in the early 1930s, without which it would have had little ability to gain political power, and IG Farben's immediate reward was guaranteed government contracts for fuel, and later profited as it moved in to take over mines and factories behind the advance of the German armies. > "By the end of 1933, the Reich agreed to buy all of the Leuna factory's output that couldn't be sold on the market, Jeffreys says. That same year, Farben donated more than 4.5 million Reich marks to Nazi Party funds." (Diarmuid Jeffreys, "Hell's Cartel", 2008)" https://web.archive.org/web/20201101015513/https://www.chicagotribune.com/news/ct-xpm-2008-08-16-0808140454-story.html https://web.archive.org/web/20201101015513/https://www.chica... They were certainly the power behind the throne, and active participants as well.

toppy 3y ago

This is so on par with Frank Herbert's prophecy from Dune: "Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them."

pixl97 3y ago

4 years ago: Why Not Just: Think of AGI Like a Corporation? https://www.youtube.com/watch?v=L5pUA3LsEaw https://www.youtube.com/watch?v=L5pUA3LsEaw >Who’s in control? Nobody. This falls in with Meditations On Moloch.

earthboundkid 3y ago

> so in a sense this is an argument that we already have rogue AI Yes. Science fiction is pretty much 100% about things that have already happened. Colonization by aliens? Colonization by Europeans. Robot slave uprisings? Human slave uprisings. AIs taking over the world? Capitalism taking over the world. Once you realize that sci-fi has to be about things that have already happened for writers to be able to write convincingly about it, it's hard to unsee. It's all just "what if the past but in the future???"

dandanua 3y ago

> Who’s in control? Nobody Greed for money and power. And for an eternal life after mind transfer, people like Musk are already dreaming about it. Surely, it's not for everyone.

piloto_ciego 3y ago

Capitalism is the AI running on all our firmware…