4 ms·

Which the refutation makes clear: it's not about intelligence, because a sufficiently powerful intelligence will just outwit its objective function in the most

Which the refutation makes clear: it's not about intelligence, because a sufficiently powerful intelligence will just outwit its objective function in the most efficient way possible (which is closer to "porn" than "destroy the Earth"). Grey goo is powerful not because it can outwit, but because it has so much brute force. On the other hand, intelligence is closer to efficiency of action: not requiring exponential time to solve something that's NP-complete, for instance. I don't think you can translate computing power like you do (a crypto miner ASIC handles a lot of bits per second, but zero FLOPS as it's all integer math; a nuclear bomb is a self-propagating reaction that alters the mass equivalent of lots of bits, but zero FLOPS). But apart from that, I agree. And I just think the "beware AI, it will become so smart that it turns into the sorcerer's apprentice" fear is misguided, because it focuses on the wrong thing. It sure makes a good narrative though! That's why it's so popular.

Isinlor 5y ago
Objective functions are just weak anthropocentric abstractions. If you take a random 110 rule or any other Turing machine how can you know its objective function? What was the objective function of abiogenesis? Can we outsmart the objective function of life? > I don't think you can translate computing power like you do. Yes, certainly human cell nucleus will not be a good general purpose computer and in this sense it's more similar to an ASIC. So, take it as a lower bound on computing power necessary to brute-force simulate the virus replication. I think it should be accurate this way. Classical computers will need at least n operations to physically copy n bits. There are also arguments that biology is very close to the thermodynamic limits of computations [0] [1]. > Here we show that the computational efficiency of translation, defined as free energy expended per amino acid operation, outperforms the best supercomputers by several orders of magnitude, and is only about an order of magnitude worse than the Landauer bound. [0] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5686401/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5686401/ [1] https://en.wikipedia.org/wiki/Landauer%27s_principle https://en.wikipedia.org/wiki/Landauer%27s_principle And yes, nuclear bombs are extremely useless as computational devices, but also extremely dangerous. They share runaway aspect with viruses, but the good thing is that nuclear bombs self-destroy. Unfortunately programs like viruses reach equilibrium with the environment and are extremely successful at keeping on existing under natural selection. There are also good reasons to believe that viruses are in a local maximum. The global maximum would be spreading to the whole universe. Overall, it does not matter if a computation is efficient as long as it has access to enough power and stars provide ridiculous amounts of power.

Isinlor 5y ago

Objective functions are just weak anthropocentric abstractions. If you take a random 110 rule or any other Turing machine how can you know its objective function? What was the objective function of abiogenesis? Can we outsmart the objective function of life? > I don't think you can translate computing power like you do. Yes, certainly human cell nucleus will not be a good general purpose computer and in this sense it's more similar to an ASIC. So, take it as a lower bound on computing power necessary to brute-force simulate the virus replication. I think it should be accurate this way. Classical computers will need at least n operations to physically copy n bits. There are also arguments that biology is very close to the thermodynamic limits of computations [0] [1]. > Here we show that the computational efficiency of translation, defined as free energy expended per amino acid operation, outperforms the best supercomputers by several orders of magnitude, and is only about an order of magnitude worse than the Landauer bound. [0] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5686401/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5686401/ [1] https://en.wikipedia.org/wiki/Landauer%27s_principle https://en.wikipedia.org/wiki/Landauer%27s_principle And yes, nuclear bombs are extremely useless as computational devices, but also extremely dangerous. They share runaway aspect with viruses, but the good thing is that nuclear bombs self-destroy. Unfortunately programs like viruses reach equilibrium with the environment and are extremely successful at keeping on existing under natural selection. There are also good reasons to believe that viruses are in a local maximum. The global maximum would be spreading to the whole universe. Overall, it does not matter if a computation is efficient as long as it has access to enough power and stars provide ridiculous amounts of power.