3 ms·

If you Google a few words you'll find the original source and context. Viruses don't replicate without host cells, there's no "nano factory" involved, and ...

If you Google a few words you'll find the original source and context. Viruses don't replicate without host cells, there's no "nano factory" involved, and ... do tell me how to implement an accurate timekeeping mechanism inside a virus. Just to clarify: Do you have deep expertise in biotech?

TeMPOraL 2y ago
> If you Google a few words you'll find the original source and context. Fair enough. Searching leads me to: https://www.lesswrong.com/posts/LfGnzX7wm6j8MGWfT/unpacking-the-dynamics-of-agi-conflict-that-suggest-the https://www.lesswrong.com/posts/LfGnzX7wm6j8MGWfT/unpacking-... discussing: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... from which the text you quote is sourced. Curiously, the relevant paragraph starts with Eliezer saying: > The concrete example I usually use here is nanotech, because there's been pretty detailed analysis of what definitely look like physically attainable lower bounds on what should be possible with nanotech, and those lower bounds are sufficient to carry the point. My lower-bound model (...) I highlight that because here Eliezer explicitly states his belief that this is very much physically possible under the rules as we know it today, whereas you accuse him of being "anti-physics and rely on the world as we know it to operate under vastly different principles than we think right now". In this "yes so" / "not so" contest, I'm inclined to take Eliezer's side, since every living thing demonstrates similar capabilities all the time. > Viruses don't replicate without host cells, there's no "nano factory" involved That's literally what a cell is, though. A nano-factory. Biology is the one existing example of molecular nanotech. > and ... do tell me how to implement an accurate timekeeping mechanism inside a virus. IDK, plagiarize the mechanism by which any one of the countless biological counters works? Not to mention, the "on a timer" part was possibly the least relevant and most replaceable piece of the scenario. (If anything, the biggest problem in this scenario is human immune system, which is ridiculously good at dealing with nanoscale threats. This makes the best bet for any rogue actor, whether AGI or human, to repurpose one of the pathogens that has been already tuned by natural selection to work on us.) BTW. the first link omits this part of the quote, which I find humorously relevant: > (Back when I was first deploying this visualization, the wise-sounding critics said "Ah, but how do you know even a superintelligence could solve the protein folding problem, if it didn't already have planet-sized supercomputers?" but one hears less of this after the advent of AlphaFold 2, for some odd reason.) > Just to clarify: Do you have deep expertise in biotech? Deep? No. Undergraduate-level in biomedical engineering, yes, plus some books and courses on genetics, because molecular biology is a topic I very much enjoy learning about.
- tdullien 2y ago
  Well, I cannot help your inclination to take Eliezer's side, but I would like to point out all the anti-physics stuff in the original article: 1) Eliezer makes up a term ("diamondoid bacteria"), and the current scientific understanding is that we have no methods to perform nanoscale manipulations of any material that would be understood as being "diamondoid". Someone else already went through the pains of comparing the fiction to current understanding of science here: https://forum.effectivealtruism.org/posts/g72tGduJMDhqR86Ns/diamondoid-bacteria-nanobots-deadly-threat-or-dead-end-a https://forum.effectivealtruism.org/posts/g72tGduJMDhqR86Ns/... The TL;DR is: There won't be anything like a Drexler nanobot thing on any realistic time horizon, if ever. 2) The described scenario involves an AI - reasoning purely from human input, without access to any empirical experiments - succeeding at building a "nanofactory" which then builds the bacteria. The author of the above article phrases it very well: "First, forget the dream of advances in theory rendering experiment unnecessary. As I explained in a previous post, the quantum equations are just way too hard to solve with 100% accuracy, so approximations are necessary, which themselves do not scale particularly well." All our theoretical understanding of everything is often a poor abstraction of reality, and it is common for even our highest-quality models for computational fluid dynamics to diverge drastically from real-world experiments. There's simply no way an AI will "reason/simulate itself through a bunch of experiments". That's not how our world and our physics work, and largely what I mean when I accuse the x-Risk crowd of being "anti-physics". The theory of building a ballpoint pen tip is very simple. The actual execution is fiendishly hard, and only a few industrialized nations have mastered it. My experience with Eliezer's writings, and a large number of x-risk adjacent people, is that they have no experience with real-world engineering or any experimental science. They simply haven't internalized that the map isn't the territory, that the real world is messy and fundamentally unpredictable. The intellectual feats they ascribe to an AI aren't far off from the AI simply finding a shortcut to calculating the trajectory of all atoms in the atmosphere and then convincing a butterfly to flap his wing at the precisely right time that in an enormous game of billards is unleashed to build a global Maxwell's demon which incinerates one half of the world. Conceivable, if you neglect that (a) you can't gather the required information and (b) you can't perform the compute required. Anyhow, my suspicion is that if the original nanobot madness didn't convince you, nothing I can write will either :-) so I think I'll excuse myself from this discussion.

TeMPOraL 2y ago

> If you Google a few words you'll find the original source and context. Fair enough. Searching leads me to: https://www.lesswrong.com/posts/LfGnzX7wm6j8MGWfT/unpacking-the-dynamics-of-agi-conflict-that-suggest-the https://www.lesswrong.com/posts/LfGnzX7wm6j8MGWfT/unpacking-... discussing: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... from which the text you quote is sourced. Curiously, the relevant paragraph starts with Eliezer saying: > The concrete example I usually use here is nanotech, because there's been pretty detailed analysis of what definitely look like physically attainable lower bounds on what should be possible with nanotech, and those lower bounds are sufficient to carry the point. My lower-bound model (...) I highlight that because here Eliezer explicitly states his belief that this is very much physically possible under the rules as we know it today, whereas you accuse him of being "anti-physics and rely on the world as we know it to operate under vastly different principles than we think right now". In this "yes so" / "not so" contest, I'm inclined to take Eliezer's side, since every living thing demonstrates similar capabilities all the time. > Viruses don't replicate without host cells, there's no "nano factory" involved That's literally what a cell is, though. A nano-factory. Biology is the one existing example of molecular nanotech. > and ... do tell me how to implement an accurate timekeeping mechanism inside a virus. IDK, plagiarize the mechanism by which any one of the countless biological counters works? Not to mention, the "on a timer" part was possibly the least relevant and most replaceable piece of the scenario. (If anything, the biggest problem in this scenario is human immune system, which is ridiculously good at dealing with nanoscale threats. This makes the best bet for any rogue actor, whether AGI or human, to repurpose one of the pathogens that has been already tuned by natural selection to work on us.) BTW. the first link omits this part of the quote, which I find humorously relevant: > (Back when I was first deploying this visualization, the wise-sounding critics said "Ah, but how do you know even a superintelligence could solve the protein folding problem, if it didn't already have planet-sized supercomputers?" but one hears less of this after the advent of AlphaFold 2, for some odd reason.) > Just to clarify: Do you have deep expertise in biotech? Deep? No. Undergraduate-level in biomedical engineering, yes, plus some books and courses on genetics, because molecular biology is a topic I very much enjoy learning about.

tdullien 2y ago
Well, I cannot help your inclination to take Eliezer's side, but I would like to point out all the anti-physics stuff in the original article: 1) Eliezer makes up a term ("diamondoid bacteria"), and the current scientific understanding is that we have no methods to perform nanoscale manipulations of any material that would be understood as being "diamondoid". Someone else already went through the pains of comparing the fiction to current understanding of science here: https://forum.effectivealtruism.org/posts/g72tGduJMDhqR86Ns/diamondoid-bacteria-nanobots-deadly-threat-or-dead-end-a https://forum.effectivealtruism.org/posts/g72tGduJMDhqR86Ns/... The TL;DR is: There won't be anything like a Drexler nanobot thing on any realistic time horizon, if ever. 2) The described scenario involves an AI - reasoning purely from human input, without access to any empirical experiments - succeeding at building a "nanofactory" which then builds the bacteria. The author of the above article phrases it very well: "First, forget the dream of advances in theory rendering experiment unnecessary. As I explained in a previous post, the quantum equations are just way too hard to solve with 100% accuracy, so approximations are necessary, which themselves do not scale particularly well." All our theoretical understanding of everything is often a poor abstraction of reality, and it is common for even our highest-quality models for computational fluid dynamics to diverge drastically from real-world experiments. There's simply no way an AI will "reason/simulate itself through a bunch of experiments". That's not how our world and our physics work, and largely what I mean when I accuse the x-Risk crowd of being "anti-physics". The theory of building a ballpoint pen tip is very simple. The actual execution is fiendishly hard, and only a few industrialized nations have mastered it. My experience with Eliezer's writings, and a large number of x-risk adjacent people, is that they have no experience with real-world engineering or any experimental science. They simply haven't internalized that the map isn't the territory, that the real world is messy and fundamentally unpredictable. The intellectual feats they ascribe to an AI aren't far off from the AI simply finding a shortcut to calculating the trajectory of all atoms in the atmosphere and then convincing a butterfly to flap his wing at the precisely right time that in an enormous game of billards is unleashed to build a global Maxwell's demon which incinerates one half of the world. Conceivable, if you neglect that (a) you can't gather the required information and (b) you can't perform the compute required. Anyhow, my suspicion is that if the original nanobot madness didn't convince you, nothing I can write will either :-) so I think I'll excuse myself from this discussion.