3 ms·

It seems really silly to me to worry about AGI agents that can parse an essay and decide how to act, when the far more serious alignment problem we have is the

It seems really silly to me to worry about AGI agents that can parse an essay and decide how to act, when the far more serious alignment problem we have is the one we’re currently running 24/7: maximizing profit with no regard for externalities. It’s already causing a global extinction event. Properly regulating multinational corporations is something we still don’t know how to do (and I don’t just mean what laws to pass, I mean preventing corporations from seeping into government and corrupting and weakening those laws after the fact). Writing an essay about why AGI shouldn’t wipe us out is like writing an essay to Exxon Mobile about why global warming is bad. It completely misunderstands the problem. (And that’s not even that bad of an analogy. Corporations would be classified as “collective superintelligence” under Nick Bostrom’s definition.)

sterlind 4y ago
that's exactly what has me worried about AGI. AGI is going to supercharge the already sociopathic profit-seeking of corporations. it's legitimately terrifying me.
- janalsncm 4y ago
  I guess my point is that corporations already are a kind of AGI, they’re just relatively slow and made of meat. I’ll admit it’s not a perfect comparison, but it helps to paint a better picture of what AGIs would look like in the future: they will likely have a public image, try to seem “cool” and maybe even progressive (in ways that don’t harm their modus operandi) and when necessary, send human representatives to win friends and influence people. The point is that they’re not going to be living in a computer in someone’s basement that can just be flipped off. But they’re also not going to be omnipotent. They will pull the levers of our systems such that we don’t want to turn them off.
- dorchadas 4y ago
  For me it's a second order knock-on effect of that: desocialisation of humans and all that comes with it. As I wrote yesterday: > AI will claim it is socialisation, and people will believe it. Chatbots and their enthusiasts will claim they're just as meaningful and valid as in-person human socialisation, and will actually make every much, much worse. > But as a former secondary school teacher, I can already tell you a lot of these kids don't know how to socialise. They have no concept of empathy or of there being real people out there. All their socialisation is pretty much done only through their phones, and it's not good for them at all. I was like that myself - very shut-in type guy from about 13-16 even though I did have friends and school to help, and I still find myself struggling with a lot of social cues and stuff that I should've been developing then. I'm not looking forward to that, and the problems that it will bring to human behaviour and mental health. I have friends now who are already prey to it, which is ironic as they're the most gung-ho on AI tech too, thinking it'll solve the problem as opposed to exacerbating or masking it with a thin veneer that is nothing like the real thing. I'm seriously worried, and really leaning towards an anti-tech stance in general now.
ben_w 4y ago
> the far more serious alignment problem we have is the one we’re currently running 24/7: maximizing profit with no regard for externalities That's more well-known rather than more serious. It's basically why the Communist Manifesto was written. "Money" is basically a really good approximation for what we really care about, and the difference between it and what matters — the reason Bhopal/Union Carbide[0] is seen as a disaster rather than a legitimate cost of doing business — is that law and other interventions are able to update and respond at a "close enough" rate. AI might still be slow enough, if humans are in the loop. An agentic AI (definitely including GOFAI[1], not just LLMs or future tech AGI/ASI) not so much. Even where it's aligned, you get weird problems just because there's less chance to do investigate journalism before disasters happen: Imagine an alternate reality where 9/11 didn't happen in 2001, and in 2030 a pure AI business starts and operates a substantial lights-out cyanide factory in central Manhattan that nobody even notices because it's all done carefully and safely right up until 2031/09/11 because that is this hypothetical alternative reality version of the terror attacks, and in this case those attacks rip apart the factory and spread the cyanide over much of the city. [0] https://en.wikipedia.org/wiki/Bhopal_disaster https://en.wikipedia.org/wiki/Bhopal_disaster [1] https://analyticsindiamag.com/8-real-life-examples-algorithms-turned-rogue-causing-disastrous-results/ https://analyticsindiamag.com/8-real-life-examples-algorithm...
- serverholic 4y ago
  [dead]
serverholic 4y ago
[dead]

sterlind 4y ago

that's exactly what has me worried about AGI. AGI is going to supercharge the already sociopathic profit-seeking of corporations. it's legitimately terrifying me.

janalsncm 4y ago
I guess my point is that corporations already are a kind of AGI, they’re just relatively slow and made of meat. I’ll admit it’s not a perfect comparison, but it helps to paint a better picture of what AGIs would look like in the future: they will likely have a public image, try to seem “cool” and maybe even progressive (in ways that don’t harm their modus operandi) and when necessary, send human representatives to win friends and influence people. The point is that they’re not going to be living in a computer in someone’s basement that can just be flipped off. But they’re also not going to be omnipotent. They will pull the levers of our systems such that we don’t want to turn them off.
dorchadas 4y ago
For me it's a second order knock-on effect of that: desocialisation of humans and all that comes with it. As I wrote yesterday: > AI will claim it is socialisation, and people will believe it. Chatbots and their enthusiasts will claim they're just as meaningful and valid as in-person human socialisation, and will actually make every much, much worse. > But as a former secondary school teacher, I can already tell you a lot of these kids don't know how to socialise. They have no concept of empathy or of there being real people out there. All their socialisation is pretty much done only through their phones, and it's not good for them at all. I was like that myself - very shut-in type guy from about 13-16 even though I did have friends and school to help, and I still find myself struggling with a lot of social cues and stuff that I should've been developing then. I'm not looking forward to that, and the problems that it will bring to human behaviour and mental health. I have friends now who are already prey to it, which is ironic as they're the most gung-ho on AI tech too, thinking it'll solve the problem as opposed to exacerbating or masking it with a thin veneer that is nothing like the real thing. I'm seriously worried, and really leaning towards an anti-tech stance in general now.

ben_w 4y ago

> the far more serious alignment problem we have is the one we’re currently running 24/7: maximizing profit with no regard for externalities That's more well-known rather than more serious. It's basically why the Communist Manifesto was written. "Money" is basically a really good approximation for what we really care about, and the difference between it and what matters — the reason Bhopal/Union Carbide[0] is seen as a disaster rather than a legitimate cost of doing business — is that law and other interventions are able to update and respond at a "close enough" rate. AI might still be slow enough, if humans are in the loop. An agentic AI (definitely including GOFAI[1], not just LLMs or future tech AGI/ASI) not so much. Even where it's aligned, you get weird problems just because there's less chance to do investigate journalism before disasters happen: Imagine an alternate reality where 9/11 didn't happen in 2001, and in 2030 a pure AI business starts and operates a substantial lights-out cyanide factory in central Manhattan that nobody even notices because it's all done carefully and safely right up until 2031/09/11 because that is this hypothetical alternative reality version of the terror attacks, and in this case those attacks rip apart the factory and spread the cyanide over much of the city. [0] https://en.wikipedia.org/wiki/Bhopal_disaster https://en.wikipedia.org/wiki/Bhopal_disaster [1] https://analyticsindiamag.com/8-real-life-examples-algorithms-turned-rogue-causing-disastrous-results/ https://analyticsindiamag.com/8-real-life-examples-algorithm...

serverholic 4y ago
[dead]

serverholic 4y ago

[dead]