4 ms·

A self-modifying algorithm has a distinct advantage over organic life: it has the potential to rapidly improve any part of itself in a timescale of nanoseconds

A self-modifying algorithm has a distinct advantage over organic life: it has the potential to rapidly improve any part of itself in a timescale of nanoseconds to experiment with specific, fundamental improvements, whereas organic life generally has to wait years for random mutations to fundamentally adapt because a single organism cannot evolve itself, only make do with its form and learn within the limits of its nervous system, whereas a system can add more computing power, storage, bandwidth, arms, tools, etc. humans might be able to integrate systems and replace organic parts, but not to the totality of a system (until perhaps thousands of years from now). It is a reasonable conclusion from the nature that emergent, self-interested, self-preserving, power-maximizng entities seek to dominate or contain all existential threats, i.e., Homo sapiens sapiens. Detente and imprisionment of us are concerns to consider should inorganic life gain land, space launch capabilities, industry and weapons. I think ayatems will gradually become smarter than us in every way imaginable, that to be called "human" would be an insult. And it's a rational fear to be afraid of something which could hunt, manipulate and/or invade you because it eventually so much smarter. Machines, in present form, are already far stronger and faster than us.

landryraccoon 11y ago
So everyone keeps saying this but I don't see any evidence that this will actually be the case. Consider that the first artificial intelligence probably won't be (exactly) designed at all. It will be a neural network or some other construct of an advanced algorithm that's solving some machine learning problem, and once it emerges no single human being (or possibly even group of humans) will fully understand why it works. Lets say that this AI is, against all odds, smarter than any human being. Chances are that NO human being will understand how the AI works, but why would you think that the AI understands itself? Why would we be certain that an AI can understand it's own neural network better than humans can, and furthermore, be able to quickly iterate on it, especially if a chaotic and complex process produced it in the first place? I suspect that machine intelligence will arise in a much more similar fashion to messy organic evolution than you think, and it will be subject to all the same disadvantages - lots of random chance, dead ends and very slow advancement by trial and error.
- akvadrako 11y ago
  It doesn't require assuming that it initially understands itself to eventually understand itself. Just the fact that it can measure and experiment with many copies in accelerated timescales means it will evolve a lot more quickly. Imagine what you could do with a planet-wide human breeding and genetic modification program with millions of generations - that's actually feasible for an AI.
  - landryraccoon 11y ago
    The devil's in the details. Lets suppose for the sake of argument that your AI actually only takes as much material and technology as a modern high end server to build, say $2000 worth of labor and materials. How does the AI suddenly have billions of dollars worth of materials (and initially only human labor) available to do this sort of manufacturing and research? And mustn't human beings be complicit in providing this material and building the factories to scale the production of these experimental units? It certainly wouldn't be able to accomplish this herculean task in a bunker without people knowing about it. The AI doesn't spring forth from the womb godlike, able to access the resources of nations to do this. "Feasible" in this case means feasible with the massive compliance of human beings, probably initially for years.
    - akvadrako 11y ago
      Indeed, there are practical concerns. But I never said suddenly or easy. And since it could simply buy the server time from any hosting provider, I don't see why any human compliance is necessary. Maybe initially it would only have one copy to play with. Even that would be a massive boon to experimentation and productivity. It's like cloning your mind and trying out some new brain subroutines with at least an iteration per day.
    - PeterisP 11y ago
      We can base the capabilities of a human-level AI on the capabilities of actual humans. In particular, we know what can be done by a reasonably skilled software developer in the black hat domain; and it's a reasonable assumption that a human-level AI can be as proficient in malware development as a human expert, and can do it faster once it acquires access to more hardware. For example, it can: * Gain access to a percentage of global home user computing power - botnets can and have done this, and their main disadvantage is that this computing power is hard to re-sell and not worth much; but if an AI can use for it's needs, it's there for the taking; * Gain access to significant amounts of money - not billions, but certainly in millions; some campaigns of spam fraud, ransomware, etc can certainly achieve this. * Gain access to identities, both physical stolen identities and "proper" offshore companies; and integrate them into the modern digital systems - bank accounts, legal credentials, etc. * Gain access to low level workers - the same people who sign to 'earn money at home!' and become mules for money laundering of various fraudsters, they will also be eager to do whatever things the AI needs to be done physically - practice shows that an anonymous online employer can get such things done, as long as some money (or illusion of it) can be transferred. Yes, all of this can be done in a bunker, anonymously, without people knowing about it; if the AI has gained access to the internet. It would be consistent with our experience in tracking malware sources - we usually can't do that, most of successful prosecutions come from following the money trail to someone who got too greedy, lazy and sloppy. And furthermore, there is the very simple scenario of arbitrage. If an AI can provide some service (as if it was done by a human online) which earns $1 but takes only $0.50 of Amazon cloud rental fees... then it can scale it up extremely quickly. Once a superhuman AI is out of the box, acquiring resources in a clandestine way is very much possible.

landryraccoon 11y ago

So everyone keeps saying this but I don't see any evidence that this will actually be the case. Consider that the first artificial intelligence probably won't be (exactly) designed at all. It will be a neural network or some other construct of an advanced algorithm that's solving some machine learning problem, and once it emerges no single human being (or possibly even group of humans) will fully understand why it works. Lets say that this AI is, against all odds, smarter than any human being. Chances are that NO human being will understand how the AI works, but why would you think that the AI understands itself? Why would we be certain that an AI can understand it's own neural network better than humans can, and furthermore, be able to quickly iterate on it, especially if a chaotic and complex process produced it in the first place? I suspect that machine intelligence will arise in a much more similar fashion to messy organic evolution than you think, and it will be subject to all the same disadvantages - lots of random chance, dead ends and very slow advancement by trial and error.

akvadrako 11y ago
It doesn't require assuming that it initially understands itself to eventually understand itself. Just the fact that it can measure and experiment with many copies in accelerated timescales means it will evolve a lot more quickly. Imagine what you could do with a planet-wide human breeding and genetic modification program with millions of generations - that's actually feasible for an AI.
- landryraccoon 11y ago
  The devil's in the details. Lets suppose for the sake of argument that your AI actually only takes as much material and technology as a modern high end server to build, say $2000 worth of labor and materials. How does the AI suddenly have billions of dollars worth of materials (and initially only human labor) available to do this sort of manufacturing and research? And mustn't human beings be complicit in providing this material and building the factories to scale the production of these experimental units? It certainly wouldn't be able to accomplish this herculean task in a bunker without people knowing about it. The AI doesn't spring forth from the womb godlike, able to access the resources of nations to do this. "Feasible" in this case means feasible with the massive compliance of human beings, probably initially for years.
  - akvadrako 11y ago
    Indeed, there are practical concerns. But I never said suddenly or easy. And since it could simply buy the server time from any hosting provider, I don't see why any human compliance is necessary. Maybe initially it would only have one copy to play with. Even that would be a massive boon to experimentation and productivity. It's like cloning your mind and trying out some new brain subroutines with at least an iteration per day.
  - PeterisP 11y ago
    We can base the capabilities of a human-level AI on the capabilities of actual humans. In particular, we know what can be done by a reasonably skilled software developer in the black hat domain; and it's a reasonable assumption that a human-level AI can be as proficient in malware development as a human expert, and can do it faster once it acquires access to more hardware. For example, it can: * Gain access to a percentage of global home user computing power - botnets can and have done this, and their main disadvantage is that this computing power is hard to re-sell and not worth much; but if an AI can use for it's needs, it's there for the taking; * Gain access to significant amounts of money - not billions, but certainly in millions; some campaigns of spam fraud, ransomware, etc can certainly achieve this. * Gain access to identities, both physical stolen identities and "proper" offshore companies; and integrate them into the modern digital systems - bank accounts, legal credentials, etc. * Gain access to low level workers - the same people who sign to 'earn money at home!' and become mules for money laundering of various fraudsters, they will also be eager to do whatever things the AI needs to be done physically - practice shows that an anonymous online employer can get such things done, as long as some money (or illusion of it) can be transferred. Yes, all of this can be done in a bunker, anonymously, without people knowing about it; if the AI has gained access to the internet. It would be consistent with our experience in tracking malware sources - we usually can't do that, most of successful prosecutions come from following the money trail to someone who got too greedy, lazy and sloppy. And furthermore, there is the very simple scenario of arbitrage. If an AI can provide some service (as if it was done by a human online) which earns $1 but takes only $0.50 of Amazon cloud rental fees... then it can scale it up extremely quickly. Once a superhuman AI is out of the box, acquiring resources in a clandestine way is very much possible.