3 ms·

A vast number if scenarios advocated by Yudkowski and adjacent people are anti-physics and rely on the world as we know it to operate under vastly different pri

A vast number if scenarios advocated by Yudkowski and adjacent people are anti-physics and rely on the world as we know it to operate under vastly different principles than we think right now. I think there's zero reason to think AIs will somehow find cheat codes to reality. Hence my "if ever". Do I believe we will eventually get auto-exploitation? Yes, I gave a lot of input to a friend of mine that (many years before the LLM craze) then had good results automating heap layout search. For certain classes of vulnerabilities and targets we will get auto-exploitation, and for some we already have them (even without AI or LLMs, unless you classify tree search as AI). Do I think we will get anything resembling the OP I replied to? No, that's pure (nonscience) fiction.

mewpmewp2 2y ago
How would you conclude it is anti physics? What's some example of something being anti physics?
- tdullien 2y ago
  "The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer. "
  - TeMPOraL 2y ago
    That's what viruses do, roughly. Complete with falling from the sky en masse. Biology and nanotech are, if you excuse my modern Tamarian... Corporate, when two pictures are shown. Pam, at her desk. I don't know where that quote is from, but without surrounding context, the only thing I can nitpick is the "diamondoid" bit.
    - mewpmewp2 2y ago
      The terminology there does appears interesting, but overall yes, it does sound like a modified virus like implementation. I would not be sure what to make of it without looking deeper into it.
    - tdullien 2y ago
      If you Google a few words you'll find the original source and context. Viruses don't replicate without host cells, there's no "nano factory" involved, and ... do tell me how to implement an accurate timekeeping mechanism inside a virus. Just to clarify: Do you have deep expertise in biotech?
      - TeMPOraL 2y ago
        > If you Google a few words you'll find the original source and context. Fair enough. Searching leads me to: https://www.lesswrong.com/posts/LfGnzX7wm6j8MGWfT/unpacking-the-dynamics-of-agi-conflict-that-suggest-the https://www.lesswrong.com/posts/LfGnzX7wm6j8MGWfT/unpacking-... discussing: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... from which the text you quote is sourced. Curiously, the relevant paragraph starts with Eliezer saying: > The concrete example I usually use here is nanotech, because there's been pretty detailed analysis of what definitely look like physically attainable lower bounds on what should be possible with nanotech, and those lower bounds are sufficient to carry the point. My lower-bound model (...) I highlight that because here Eliezer explicitly states his belief that this is very much physically possible under the rules as we know it today, whereas you accuse him of being "anti-physics and rely on the world as we know it to operate under vastly different principles than we think right now". In this "yes so" / "not so" contest, I'm inclined to take Eliezer's side, since every living thing demonstrates similar capabilities all the time. > Viruses don't replicate without host cells, there's no "nano factory" involved That's literally what a cell is, though. A nano-factory. Biology is the one existing example of molecular nanotech. > and ... do tell me how to implement an accurate timekeeping mechanism inside a virus. IDK, plagiarize the mechanism by which any one of the countless biological counters works? Not to mention, the "on a timer" part was possibly the least relevant and most replaceable piece of the scenario. (If anything, the biggest problem in this scenario is human immune system, which is ridiculously good at dealing with nanoscale threats. This makes the best bet for any rogue actor, whether AGI or human, to repurpose one of the pathogens that has been already tuned by natural selection to work on us.) BTW. the first link omits this part of the quote, which I find humorously relevant: > (Back when I was first deploying this visualization, the wise-sounding critics said "Ah, but how do you know even a superintelligence could solve the protein folding problem, if it didn't already have planet-sized supercomputers?" but one hears less of this after the advent of AlphaFold 2, for some odd reason.) > Just to clarify: Do you have deep expertise in biotech? Deep? No. Undergraduate-level in biomedical engineering, yes, plus some books and courses on genetics, because molecular biology is a topic I very much enjoy learning about.

mewpmewp2 2y ago

How would you conclude it is anti physics? What's some example of something being anti physics?

tdullien 2y ago
"The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer. "
- TeMPOraL 2y ago
  That's what viruses do, roughly. Complete with falling from the sky en masse. Biology and nanotech are, if you excuse my modern Tamarian... Corporate, when two pictures are shown. Pam, at her desk. I don't know where that quote is from, but without surrounding context, the only thing I can nitpick is the "diamondoid" bit.
  - mewpmewp2 2y ago
    The terminology there does appears interesting, but overall yes, it does sound like a modified virus like implementation. I would not be sure what to make of it without looking deeper into it.
  - tdullien 2y ago
    If you Google a few words you'll find the original source and context. Viruses don't replicate without host cells, there's no "nano factory" involved, and ... do tell me how to implement an accurate timekeeping mechanism inside a virus. Just to clarify: Do you have deep expertise in biotech?
    - TeMPOraL 2y ago
      > If you Google a few words you'll find the original source and context. Fair enough. Searching leads me to: https://www.lesswrong.com/posts/LfGnzX7wm6j8MGWfT/unpacking-the-dynamics-of-agi-conflict-that-suggest-the https://www.lesswrong.com/posts/LfGnzX7wm6j8MGWfT/unpacking-... discussing: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... from which the text you quote is sourced. Curiously, the relevant paragraph starts with Eliezer saying: > The concrete example I usually use here is nanotech, because there's been pretty detailed analysis of what definitely look like physically attainable lower bounds on what should be possible with nanotech, and those lower bounds are sufficient to carry the point. My lower-bound model (...) I highlight that because here Eliezer explicitly states his belief that this is very much physically possible under the rules as we know it today, whereas you accuse him of being "anti-physics and rely on the world as we know it to operate under vastly different principles than we think right now". In this "yes so" / "not so" contest, I'm inclined to take Eliezer's side, since every living thing demonstrates similar capabilities all the time. > Viruses don't replicate without host cells, there's no "nano factory" involved That's literally what a cell is, though. A nano-factory. Biology is the one existing example of molecular nanotech. > and ... do tell me how to implement an accurate timekeeping mechanism inside a virus. IDK, plagiarize the mechanism by which any one of the countless biological counters works? Not to mention, the "on a timer" part was possibly the least relevant and most replaceable piece of the scenario. (If anything, the biggest problem in this scenario is human immune system, which is ridiculously good at dealing with nanoscale threats. This makes the best bet for any rogue actor, whether AGI or human, to repurpose one of the pathogens that has been already tuned by natural selection to work on us.) BTW. the first link omits this part of the quote, which I find humorously relevant: > (Back when I was first deploying this visualization, the wise-sounding critics said "Ah, but how do you know even a superintelligence could solve the protein folding problem, if it didn't already have planet-sized supercomputers?" but one hears less of this after the advent of AlphaFold 2, for some odd reason.) > Just to clarify: Do you have deep expertise in biotech? Deep? No. Undergraduate-level in biomedical engineering, yes, plus some books and courses on genetics, because molecular biology is a topic I very much enjoy learning about.