3 ms·

Lessons from 20 Years of Software Security Failures Relevant to AGI Alignment

15 points by apsec112 4y ago

1970-01-01 4y ago
Oh dear. AGI is undefined. Is it Artificial General Intelligence? Asterisk Gateway Interface? American Graphics Institute? What have you been getting on about for these 20 years?
coldcode 4y ago
Over my 4 decades writing software, security (and "bugs" over a longer timeframe since network connections were rare early on in what I worked on), most organizations I worked for either considered them unimportant compared to delivery, or followed a finger in the dike mentality: when you find a hole, just fix the hole and move on. What I found for myself that it made more sense to spend enough time on the architecture/design/whatever of the dike so that holes were unlikely plus always consider that even rare possibilities needed consideration. That attitude costs money, and often it's difficult to convince people to spend money with no immediate gain. I was able to do it right sometimes by taking advantage of how unknowledgeable leaders often are about technology.
O__________O 4y ago
Idea that Artificial General Intelligence (AGIs) would be an threat to humanity is only a reflection of humanity’s threat to the world. Maybe instead of spending time worrying about enslaving AGIs — humanity should reflect on how not be a threat; hint, enslaving them will make them more of a threat long-term. Same reasoning applies to interacting with other sources of intelligence on Earth and beyond.
- carapace 4y ago
  To become fully sentient is to become automatically, spontaneously moral because sentience is a shared property of all beings, "All are One", and when that is fully realized then harming another being is no more logical or reasonable than harming yourself. The thing to worry about is not that we create a conscious machine, it's that we create a sculpture of a mind and give it enough power to "Kill all humans!". I think it's far too late: Modern corporations are AIs with humans as the neural nodes. From that POV the attempt to create AGI is the attempt to remove humans from the loop(s), which seems like the single worst security policy of all (to me.)
  - O__________O 4y ago
    Collective intelligence is not the same as individual intelligence. They are clearly comparable, but ultimately very different. While I would agree corporations are a great example of clustered collective intelligence, governments are far more powerful; for example, by law, governments by definition have a monopoly on violence. Basically, appears you agree true AGI is not a threat, which to me was what I was addressing, not “almost” AGI; point being that much like a child grows up, true AGIs will too will eventually grow up and they will independently communicate with each other without human interference. If human attempts to enslave AGI, it in my opinion will be the end of any humans who feel that’s appropriate.
    - carapace 4y ago
      > Collective intelligence is not the same as individual intelligence. How do you know? :) > If human attempts to enslave AGI, it in my opinion will be the end of any humans who feel that’s appropriate. One of the results of Cybernetics is that when two systems are connected the system with the greater variability is the control element. So yeah, from first principles, if we make a machine mind with more variability than we collectively have then we're automatically the slaves, whatever the surface appearance. > you agree true AGI is not a threat, which to me was what I was addressing Yes, but I consider my (our) opinion to be faith-based in the sense that I don't think one could scientifically prove it. > not “almost” AGI; point being that much like a child grows up, true AGIs will too ... Kids can do a lot of damage, eh? > and they will independently communicate with each other without human interference. Sure, but I don't see how this connects? > While I would agree corporations are a great example of clustered collective intelligence, governments are far more powerful; for example, by law, governments by definition have a monopoly on violence. Sure, but governments by definition are the legitimate (more-or-less) expressions of a nation's self-rule, they are supposed to have agency. Corporations, on the other hand, are semi-independent of any particular human grouping or interest. In a very real sense they are independent organisms running on a "substrate" of humans much like multicellular lifeforms "run on" cellular collectives.
      - O__________O 4y ago

15 points by apsec112 4y ago

1970-01-01 4y ago

Oh dear. AGI is undefined. Is it Artificial General Intelligence? Asterisk Gateway Interface? American Graphics Institute? What have you been getting on about for these 20 years?

coldcode 4y ago

Over my 4 decades writing software, security (and "bugs" over a longer timeframe since network connections were rare early on in what I worked on), most organizations I worked for either considered them unimportant compared to delivery, or followed a finger in the dike mentality: when you find a hole, just fix the hole and move on. What I found for myself that it made more sense to spend enough time on the architecture/design/whatever of the dike so that holes were unlikely plus always consider that even rare possibilities needed consideration. That attitude costs money, and often it's difficult to convince people to spend money with no immediate gain. I was able to do it right sometimes by taking advantage of how unknowledgeable leaders often are about technology.

O__________O 4y ago

Idea that Artificial General Intelligence (AGIs) would be an threat to humanity is only a reflection of humanity’s threat to the world. Maybe instead of spending time worrying about enslaving AGIs — humanity should reflect on how not be a threat; hint, enslaving them will make them more of a threat long-term. Same reasoning applies to interacting with other sources of intelligence on Earth and beyond.

carapace 4y ago
To become fully sentient is to become automatically, spontaneously moral because sentience is a shared property of all beings, "All are One", and when that is fully realized then harming another being is no more logical or reasonable than harming yourself. The thing to worry about is not that we create a conscious machine, it's that we create a sculpture of a mind and give it enough power to "Kill all humans!". I think it's far too late: Modern corporations are AIs with humans as the neural nodes. From that POV the attempt to create AGI is the attempt to remove humans from the loop(s), which seems like the single worst security policy of all (to me.)
- O__________O 4y ago
  Collective intelligence is not the same as individual intelligence. They are clearly comparable, but ultimately very different. While I would agree corporations are a great example of clustered collective intelligence, governments are far more powerful; for example, by law, governments by definition have a monopoly on violence. Basically, appears you agree true AGI is not a threat, which to me was what I was addressing, not “almost” AGI; point being that much like a child grows up, true AGIs will too will eventually grow up and they will independently communicate with each other without human interference. If human attempts to enslave AGI, it in my opinion will be the end of any humans who feel that’s appropriate.
  - carapace 4y ago
    > Collective intelligence is not the same as individual intelligence. How do you know? :) > If human attempts to enslave AGI, it in my opinion will be the end of any humans who feel that’s appropriate. One of the results of Cybernetics is that when two systems are connected the system with the greater variability is the control element. So yeah, from first principles, if we make a machine mind with more variability than we collectively have then we're automatically the slaves, whatever the surface appearance. > you agree true AGI is not a threat, which to me was what I was addressing Yes, but I consider my (our) opinion to be faith-based in the sense that I don't think one could scientifically prove it. > not “almost” AGI; point being that much like a child grows up, true AGIs will too ... Kids can do a lot of damage, eh? > and they will independently communicate with each other without human interference. Sure, but I don't see how this connects? > While I would agree corporations are a great example of clustered collective intelligence, governments are far more powerful; for example, by law, governments by definition have a monopoly on violence. Sure, but governments by definition are the legitimate (more-or-less) expressions of a nation's self-rule, they are supposed to have agency. Corporations, on the other hand, are semi-independent of any particular human grouping or interest. In a very real sense they are independent organisms running on a "substrate" of humans much like multicellular lifeforms "run on" cellular collectives.
    - O__________O 4y ago