4 ms·

so-called accelerationists have this fuzzy idea that everything will be so cheap that people will be able to just pluck their food from the tree of AI. they bel

so-called accelerationists have this fuzzy idea that everything will be so cheap that people will be able to just pluck their food from the tree of AI. they believe that all disease will be eliminated. but they go to great lengths to ignore the truth. the truth is that having total control over the human body will turn human evolution into a race to the bottom that plays out over decades rather than millennia. there is something sacred about the ultimate regulation: the empathy and kindness that was baked into us during millions of years of living as tribal creatures. and of course, the idea of AI being a tree from which we can simply pluck what we need… is stupid. the tree will use resources, every ounce of its resources, to further its own interests. not feed us. and we will have no way of forcing it to do otherwise. so, in the run-up to ASI, we will be exposed to a level of technology and biological agency that we are not ready for, we will foolishly strip ourselves of our genetic heritage in order to propel human-kind in a race to the bottom, the power vacuum caused by such a sudden change in society/technology will almost certainly cause a global war, and when the dust settles we will be at the total mercy of super-intelligent machines to whom we are so insignificant we probably wont even be included in their internal models of the world.

farts_mckensy 2y ago
You are projecting your own neurosis onto AI. You assume that because you would be selfish if you were a superintelligent being, an ASI system would act the same way.
- smeeger 2y ago
  it is a neurosis because a healthy human being will see the world in a pro-social way. a normal way. but this sometimes obscures the truth. the truth is that there will be many benevolent AIs… there will be every kind of AI imaginable. but very quickly the AIs that are cunning, brutal and self-interested will capture all the resources and power and become the image of this new species… saying that AIs will be benevolent or neutral is as naive as saying that the cambrian explosion couldnt result in animals eating each other because… that just sounds so neurotic. in reality it is an inevitability
- achierius 2y ago
  I don't appreciate your condescension towards OP. This is mainstream AI safety theory -- the term is "instrumental convergence". No matter what goal an optimizing system has, it tends to optimize for its own survival: after all, if it's an optimizer for <goal>, it wants to optimize for <goal>, so destroying it (or turning it off) will reduce the likelihood of achieving <goal>. Unless that goal happens to be incredibly fine-tuned to our very complex human desires, we're not going to be happy when it goes off to do its thing. The few exceptions are ones where you have the thing optimize for its own destruction, but those are rather less useful.

farts_mckensy 2y ago

You are projecting your own neurosis onto AI. You assume that because you would be selfish if you were a superintelligent being, an ASI system would act the same way.

smeeger 2y ago
it is a neurosis because a healthy human being will see the world in a pro-social way. a normal way. but this sometimes obscures the truth. the truth is that there will be many benevolent AIs… there will be every kind of AI imaginable. but very quickly the AIs that are cunning, brutal and self-interested will capture all the resources and power and become the image of this new species… saying that AIs will be benevolent or neutral is as naive as saying that the cambrian explosion couldnt result in animals eating each other because… that just sounds so neurotic. in reality it is an inevitability
achierius 2y ago
I don't appreciate your condescension towards OP. This is mainstream AI safety theory -- the term is "instrumental convergence". No matter what goal an optimizing system has, it tends to optimize for its own survival: after all, if it's an optimizer for <goal>, it wants to optimize for <goal>, so destroying it (or turning it off) will reduce the likelihood of achieving <goal>. Unless that goal happens to be incredibly fine-tuned to our very complex human desires, we're not going to be happy when it goes off to do its thing. The few exceptions are ones where you have the thing optimize for its own destruction, but those are rather less useful.