3 ms·

> The real threat posed by AI is one that all of us face everyday: bad software design. Is it likely that an unintended consequence will cause an AI to launch n

> The real threat posed by AI is one that all of us face everyday: bad software design. Is it likely that an unintended consequence will cause an AI to launch nuclear missiles, release toxic chemicals or shut down the global financial markets? Yes, pretty likely. AI is not the kind of thing that can be designed. They tried that route, way back in the sixties, and just failed miserably. Realistically, AI will come from some combination of genetic algorithms, training neural networks, and so on. Now, yes, it could fail, but not in the same way software as we know it fails. No, AI failure would be more similar to human failure. That is still worrying, but no more than hiring the wrong people would be, for example, and you would have better ways to evaluate them.

PeterisP 12y ago
AI failure would be a bit different than human failure. If a powerful human is obsessed and 'fails' then at worst he gathers some other people, successfully creates an evil empire and dies after a few decades. Once a powerful AI is obsessed and 'fails', then it can replace as much of the world as it wants with itself, and lives on forever.
- breuleux 12y ago
  Could it? Let's take a few steps back, here. * You're assuming the AI can copy itself, but this is a dubious assumption. As far as I know, none of the AI algorithms at the forefront of research provide a fraction of the data the AI would need to copy itself. Being able to copy yourself is not a property that comes with running on a computer and I'm quite positive strong AI, when it emerges, will lack this capability to any meaningful extent. Worse yet, data copiability is ultimately a hardware property, and it requires a way to export a snapshot of one's internal state all through the surface. That's not actually efficient design and one has to account for the possibility that AI would run on hardware that makes copies physically impossible. Locality of information minimizes distance, and this is key to efficiency. The only reason our computers architectures work the way they do is that we need them to, but AI in a production setting is not conventional software and does not need to bend to silly copiability requirements. * You're assuming it would have anywhere to copy itself on. If it's running on a billion dollars' worth of hardware, well, it can't just copy itself on user grade computers and expect to gain much out of it. I personally tend to believe that churning out new AI brains from scratch states will yield superior results to copying pre-trained AI or to "exponentially self-improving AI". If nature is to be believed, improvement often requires cycling through clean slates (e.g. birth); software development also suggests that same idea, that sometimes if you want better software you just have to rewrite it. Honestly, it's kind of a rule in general optimization.
  - TheLoneWolfling 12y ago
    It is good to not be the only person to say this. One other thing to take note of is that even if you somehow manage to copy the software state, that may not (probably won't) be enough - there are many things that self-improving software may end up unwittingly relying on that cannot be transferred between different pieces of hardware. (CPU temperature variation? Fanspeed? Webcam static? Order of race conditions between CPUs? Exact amounts of time before things are fetched from disk (or even RAM?)? Network access delays?)

PeterisP 12y ago

AI failure would be a bit different than human failure. If a powerful human is obsessed and 'fails' then at worst he gathers some other people, successfully creates an evil empire and dies after a few decades. Once a powerful AI is obsessed and 'fails', then it can replace as much of the world as it wants with itself, and lives on forever.

breuleux 12y ago
Could it? Let's take a few steps back, here. * You're assuming the AI can copy itself, but this is a dubious assumption. As far as I know, none of the AI algorithms at the forefront of research provide a fraction of the data the AI would need to copy itself. Being able to copy yourself is not a property that comes with running on a computer and I'm quite positive strong AI, when it emerges, will lack this capability to any meaningful extent. Worse yet, data copiability is ultimately a hardware property, and it requires a way to export a snapshot of one's internal state all through the surface. That's not actually efficient design and one has to account for the possibility that AI would run on hardware that makes copies physically impossible. Locality of information minimizes distance, and this is key to efficiency. The only reason our computers architectures work the way they do is that we need them to, but AI in a production setting is not conventional software and does not need to bend to silly copiability requirements. * You're assuming it would have anywhere to copy itself on. If it's running on a billion dollars' worth of hardware, well, it can't just copy itself on user grade computers and expect to gain much out of it. I personally tend to believe that churning out new AI brains from scratch states will yield superior results to copying pre-trained AI or to "exponentially self-improving AI". If nature is to be believed, improvement often requires cycling through clean slates (e.g. birth); software development also suggests that same idea, that sometimes if you want better software you just have to rewrite it. Honestly, it's kind of a rule in general optimization.
- TheLoneWolfling 12y ago
  It is good to not be the only person to say this. One other thing to take note of is that even if you somehow manage to copy the software state, that may not (probably won't) be enough - there are many things that self-improving software may end up unwittingly relying on that cannot be transferred between different pieces of hardware. (CPU temperature variation? Fanspeed? Webcam static? Order of race conditions between CPUs? Exact amounts of time before things are fetched from disk (or even RAM?)? Network access delays?)