3 ms·

It's just a numbers game. Remember that last year[1] a crypto scammer convinced a bank CEO to embezzle $47mm, leading to the bank failing and two years in pris

It's just a numbers game. Remember that last year[1] a crypto scammer convinced a bank CEO to embezzle $47mm, leading to the bank failing and two years in prison for the CEO. There's always someone out there who can be tricked, even tricked into large-scale mistakes that will end their career. [1]: https://www.fbi.gov/news/stories/fbi-recovers-8-million-swindled-from-failed-kansas-banks-small-town-investors https://www.fbi.gov/news/stories/fbi-recovers-8-million-swin...

strken 1y ago
I mean, sure, you will eventually be able to convince someone, somewhere to do anything, but the whole thought experiment of a superintelligent AI in a box is about the AI convincing one specific human to let it go using only superhuman charisma and what it learns via text messaging back and forth. That seems...a lot less likely, especially given limited or zero information about the outside world and a target who knows you're an AI in a box.
- lcnPylGDnU4H9OF 1y ago
  This theoretical opponent would start to attack the motive of the person. I think the superhuman charisma could be enough to convince someone to quit their job for any number of reasons and, oh, since they're leaving anyway don't they want to see what happens if they let the AI out of the box?
  - strken 1y ago
    I question whether such a thing as superhuman charisma exists. Or if it exists, whether it can be used over chat without access to body language and human empathy, against someone who refuses to engage from the outset, and without side channels for gaining information about the human. I get that not pressing the big red button might be a problem for people who think Roko's Basilisk is an actual threat and have a culture of considering seemingly absurd ideas, but I question whether it's a problem for everyone else.
    - jrowen 1y ago
      I think the biggest issue with the thought experiment is that there's never going to be one person sitting there that can press a button that irreversibly dooms humanity. There's never even an attempt at a logical sequence of events that leads from the moment of that breach to our complete doom. It's not inconceivable but I feel like these fears need to be grounded in a plausible "here's how it could actually happen." What prevents the combined abilities of the human race from shutting it down? AI isn't magic, it is still confined to the pretty fragile and narrow bounds of digital technology.

strken 1y ago

I mean, sure, you will eventually be able to convince someone, somewhere to do anything, but the whole thought experiment of a superintelligent AI in a box is about the AI convincing one specific human to let it go using only superhuman charisma and what it learns via text messaging back and forth. That seems...a lot less likely, especially given limited or zero information about the outside world and a target who knows you're an AI in a box.

lcnPylGDnU4H9OF 1y ago
This theoretical opponent would start to attack the motive of the person. I think the superhuman charisma could be enough to convince someone to quit their job for any number of reasons and, oh, since they're leaving anyway don't they want to see what happens if they let the AI out of the box?
- strken 1y ago
  I question whether such a thing as superhuman charisma exists. Or if it exists, whether it can be used over chat without access to body language and human empathy, against someone who refuses to engage from the outset, and without side channels for gaining information about the human. I get that not pressing the big red button might be a problem for people who think Roko's Basilisk is an actual threat and have a culture of considering seemingly absurd ideas, but I question whether it's a problem for everyone else.
  - jrowen 1y ago
    I think the biggest issue with the thought experiment is that there's never going to be one person sitting there that can press a button that irreversibly dooms humanity. There's never even an attempt at a logical sequence of events that leads from the moment of that breach to our complete doom. It's not inconceivable but I feel like these fears need to be grounded in a plausible "here's how it could actually happen." What prevents the combined abilities of the human race from shutting it down? AI isn't magic, it is still confined to the pretty fragile and narrow bounds of digital technology.