5 ms·

Wow, what a perfect description of why their probability-logic leads to silly beliefs. I've been wondering how to argue within their frame for a while, and her

Wow, what a perfect description of why their probability-logic leads to silly beliefs. I've been wondering how to argue within their frame for a while, and here's what I've come up with: Is the likelihood that aliens exist, are unfriendly, and AGI will help us beat them higher or lower than the likelihood that the AGI itself that we develop is unfriendly to us and wants to FOOM us? Show your work.

idunnoman1222 2y ago
They think they can predict the future by extension know what’s good for us. If they could choose you wouldn’t get a vote.
shadowgovt 2y ago
Much of philosophy throughout history seems to operate this way. I think philosophy is a noble pursuit, but it's worth noting how often people drew very broad conclusions, and then acted on them, from not very much data. Consider the dozens of theories of the constitution of the world from the time of the Greek thinkers (even the atomic theory doesn't look very much at all like atoms as we now understand them), or the myriad examples of political philosophies that ran up against people simply not acting the way the philosophy needed them to act to cohere. The investigation of possibility is laudable, but a healthy and regular dose of evidence is important.
- suddenlybananas 2y ago
  The idea that you need evidence to justify your beliefs is a philosophical position.
  - Seattle3503 2y ago
    It's a pretty good one.
    - somenameforme 2y ago
      Anything about the self bumps into an immediate problem here. For instance I cannot prove to you that I'm conscious and not simply an automaton who's not actually thinking. My evidence for such is strictly personal - I can personally testify to my own experience of consciousness, but you have no reason to believe me since that's not evidence. And in fact even for myself - perception is not necessarily valid evidence since perception can be distorted. If I am in a compelling VR game I might be more than willing to swear to the fact that I'm flying (if I wasn't otherwise aware of the situation) - while you simply look at me acting a fool standing still while vigorously flapping my arms.
      - shadowgovt 2y ago
        ... so at some point, one realizes one has pondered one's way into untestables and goes back to living. Or doesn't, I guess, and then gets kept up at night anxious about the notion that in some as-yet-unrealized future, an AI is forever torturing an identical copy of oneself that one cannot possibly ever meet. The programming analogy to this kind of philosophy is writing design docs (or building a class hierarchy of abstracts) without ever writing implementation. Lots of work, but why should anyone outside the room care?
        somenameforme 2y ago
        It contradicts the ideals of an evidence based system of values. Most of what we believe we believe because we think it is right, and there's always , more or less, viable arguments for most of any remotely reasonable view. And this applies to all people. For instance it was none other than Max Planck that observed, "Science progresses one funeral at a time." I also think this is for the best. If one looks at the evidence of the skies it's indisputable that humanity lies at the center of the cosmos, with everything revolving around us - which, in turn, naturally leads into religious narratives around Earth. It's only thanks to these weirdos that adopted quite absurd sounding (for the time) systems of views and values, completely without meaningful evidence at first, that we were able to gradually expand, and correct, our understanding of the universe. And this doesn't just apply to the ancient past. Einstein's observation that the speed of light stays fixed while the rate of passage of time itself is variable, to enable the former, sounds so utterly and absolutely absurd. In many ways the real challenge in breaking through to relativity wasn't the math or anything like that (which, in fact, Einstein himself lacked when first developing the concept) but accepting that a concept that sounds so wrong might actually be right.
  - shadowgovt 2y ago
    Excellent point. And it shines a light directly on what I'm saying. There's a great line from the Hitchhiker's Guide to the Galaxy: """"Oh, that was easy," says Man, and for an encore goes on to prove that black is white and gets himself killed on the next zebra crossing.”""" ... and, of course, the philosopher can argue that we are all of us tumbling through a Pachinko-machine of parallel universes, and it is only our constrained perception that suggests Man is dead; in a parallel universe, Man is just fine and living their days in the grey universe that we, through some as-yet-unphilosophized-but-don't-worry-we're-thinking-hard-about-it process, cannot exist in. ... but does that matter over-much if the observable is that ignoring zebra crossings gets you crossed out of this universe?
- implements 2y ago
  > Much of philosophy throughout history seems to operate this way. “Philosophy is poor at revealing truths but excellent at revealing falsehoods (or at least unsupported arguments)” was the main lesson I took from informally studying it.
- yawboakye 2y ago
  very few philosophers dared to live by their theories. the famous failures of aristotle (in the case of alexander) and plato in syracuse (where he saw firsthand that the philosopher-king is at best a book character) are good examples. seneca didn’t live stoically: he was avaricious and didn’t bother to incite civil war over unpaid debt, if ancient sources are to be believed. he failed horribly with nero, who later instructed him to commit suicide for treasonous crimes. again, if ancient sources are to be believed, he fumbled his suicide out of raw fear. the cynics, though, made a good life but that’s not because they had a better philosophy. it’s because cynicism is base/primitive logic available to the brute as well as the civilized man.
  - lukan 2y ago
    And then there is Rousseau, who wrote books about raising children. But raising his own was too much.
    - yawboakye 2y ago
      rousseau is a fraud of the first order.
      - lukan 2y ago
        Enlightened fraud
    - shadowgovt 2y ago
      I actually have on my desk, right now, Mary Wollstonecraft's "A Vindication of the Rights of Woman." To summarize one of her points from memory, she basically lays into Rousseau about having a lot of opinions on domestic life for someone who clearly doesn't even know how to set foot in a kitchen.
waveBidder 2y ago
the term you're looking for is pascal's mugging, and it originates from within rationalism
- IshKebab 2y ago
  It seems a bit nonsense to me. > The mugger argues back that for any low but strictly greater than 0 probability of being able to pay back a large amount of money (or pure utility) there exists a finite amount that makes it rational to take the bet. This is a basic logic error. It ignores the very obvious fact that increasing the reward amount decreases the probability that it will be returned. E.g. if the probability of the reward R being returned is (0.5/R) we get "a low by strictly greater than 0 probability", and for that probability there is a (different) finite reward that would make it rational to take the bet, but it's not R. This is even simpler and more stupid than the "proofs" that 0=1. It does not change my opinion that philosophers (and lesswrong) are idiots.
  - stuartjohnson12 2y ago
    That's what the person responding to meant - attempts to make human systems "rational" often involves simplifying dependent probabilities and presenting them as independent. The rationalist community both frequently makes this reasoning error and is aware they frequently make this error, and coined this term to refer to the category of reasoning mistake.
    - IshKebab 2y ago
      > coined this term to refer to the category of reasoning mistake. That's not at all what the Wikipedia article for it says. It presents it as an interesting paradox with several potential (and incorrect!) "remedies" rather than a category of basic logical errors.
      - close04 2y ago
        The “quadrillion days of happiness” offered to a rational person gives away that such allegories are anthropomorphized just for the sake of presentation. For the sake of what the philosophers mean, you should probably imagine this as an algorithm running on a machine (no AGI). It’s a mental tease, not a manual on how to react when faced with a mugger who forgot his weapon at home but has an interesting proposition to make. Similarly the trolley problem isn’t really about real humans in that situation, or else the correct answer would always be “do nothing”. It’s what the comment here [0] says. If you try to analyze everything purely rationally it will lead to dark corners of misunderstanding and madness. [0] https://news.ycombinator.com/item?id=42902931 https://news.ycombinator.com/item?id=42902931
michaelkeenan 2y ago
AGI would be extremely helpful in navigating clashes with aliens, but taking the time to make sure it's safe is very unlikely to make a difference to whether it's ready in time. Rationalists want AGI to be built, and they're generally very excited about it, e.g. many of them work at Anthropic. They just don't want a Move Fast and Break Things pace of development.
nkrisc 2y ago
It’s pointless. They aren’t rational. Any argument you come up with that contradicts their personal desires will be successfully “reasoned” away by them because they want it to be. Your mistake was ever thinking they had a rational thought to begin with, they think they are infallible.
- HeatrayEnjoyer 2y ago
  "Widespread robots that make their own decisions autonomously will probably be very bad for humans if they make decisions that aren't in our interest" isn't really that much of a stretch is it? If we were going slower maybe it would seem more theoretical. But there are multiple Manhattan-Project-level or (often, much) larger efforts ongoing to explicitly create software and robotic hardware that makes decisions and takes actions without any human in the loop. We don't need some kind of 10000 IQ god intelligence if a glitch token causes the majority of the labor force to suddenly and collectively engage in sabotage.
  - NoMoreNicksLeft 2y ago
    None of those projects are even heading in the direction of "AGI". The state of AI today is something akin to what science fiction would have called an "oracle", a devise that can answer questions intelligently or seemingly-intelligently but otherwise isn't intelligent at all: it can't learn, has no agency, does nothing new. Even if that can be scaled up indefinitely, there's no reason to believe that it will ever become an AGI. If any of this can make decisions in a way that a human can, then I would start to question what human decision-making really amounts to.
    - rachofsunshine 2y ago
      For what it's worth, as the person at the top of this thread, I actually do take AI risk pretty seriously. Not in a singulatarian sense, but in the sense that I would be quite surprised if AI weren't capable of this stuff in ten years. Even the oracle version is already really dangerous in the wrong hands. A totalitarian government doesn't need to have someone listening to a few specific dissidents if they can have an AI transcriber write down every word of every phone conversation in the country, for example. And while it's certainly not error-proof, asking an LLM to do something like "go through this list of conversations and flag anything that sounds like anti-government sentiment" is going to get plenty of hits, too.
throwaway4aday 2y ago
It seems that you didn't understand the main point of the exposition. I'll summarize the ops comment a bit further. Points 1 and 2 only explain how they are able to erroneously justify their absurd beliefs, they don't explain why they hold those beliefs. Points 3 through 5 are the heart of the matter; egotistical and charismatic (to some types of people) leaders, open minded, freethinking and somewhat weird or marginalized people searching for meaning plus a way for them all to congregate around some shared interests. TLDR: perfect conditions for one or more cults to form.

idunnoman1222 2y ago

They think they can predict the future by extension know what’s good for us. If they could choose you wouldn’t get a vote.

shadowgovt 2y ago

Much of philosophy throughout history seems to operate this way. I think philosophy is a noble pursuit, but it's worth noting how often people drew very broad conclusions, and then acted on them, from not very much data. Consider the dozens of theories of the constitution of the world from the time of the Greek thinkers (even the atomic theory doesn't look very much at all like atoms as we now understand them), or the myriad examples of political philosophies that ran up against people simply not acting the way the philosophy needed them to act to cohere. The investigation of possibility is laudable, but a healthy and regular dose of evidence is important.

suddenlybananas 2y ago
The idea that you need evidence to justify your beliefs is a philosophical position.
- Seattle3503 2y ago
  It's a pretty good one.
  - somenameforme 2y ago
    Anything about the self bumps into an immediate problem here. For instance I cannot prove to you that I'm conscious and not simply an automaton who's not actually thinking. My evidence for such is strictly personal - I can personally testify to my own experience of consciousness, but you have no reason to believe me since that's not evidence. And in fact even for myself - perception is not necessarily valid evidence since perception can be distorted. If I am in a compelling VR game I might be more than willing to swear to the fact that I'm flying (if I wasn't otherwise aware of the situation) - while you simply look at me acting a fool standing still while vigorously flapping my arms.
    - shadowgovt 2y ago
      ... so at some point, one realizes one has pondered one's way into untestables and goes back to living. Or doesn't, I guess, and then gets kept up at night anxious about the notion that in some as-yet-unrealized future, an AI is forever torturing an identical copy of oneself that one cannot possibly ever meet. The programming analogy to this kind of philosophy is writing design docs (or building a class hierarchy of abstracts) without ever writing implementation. Lots of work, but why should anyone outside the room care?
      - somenameforme 2y ago
        It contradicts the ideals of an evidence based system of values. Most of what we believe we believe because we think it is right, and there's always , more or less, viable arguments for most of any remotely reasonable view. And this applies to all people. For instance it was none other than Max Planck that observed, "Science progresses one funeral at a time." I also think this is for the best. If one looks at the evidence of the skies it's indisputable that humanity lies at the center of the cosmos, with everything revolving around us - which, in turn, naturally leads into religious narratives around Earth. It's only thanks to these weirdos that adopted quite absurd sounding (for the time) systems of views and values, completely without meaningful evidence at first, that we were able to gradually expand, and correct, our understanding of the universe. And this doesn't just apply to the ancient past. Einstein's observation that the speed of light stays fixed while the rate of passage of time itself is variable, to enable the former, sounds so utterly and absolutely absurd. In many ways the real challenge in breaking through to relativity wasn't the math or anything like that (which, in fact, Einstein himself lacked when first developing the concept) but accepting that a concept that sounds so wrong might actually be right.
- shadowgovt 2y ago
  Excellent point. And it shines a light directly on what I'm saying. There's a great line from the Hitchhiker's Guide to the Galaxy: """"Oh, that was easy," says Man, and for an encore goes on to prove that black is white and gets himself killed on the next zebra crossing.”""" ... and, of course, the philosopher can argue that we are all of us tumbling through a Pachinko-machine of parallel universes, and it is only our constrained perception that suggests Man is dead; in a parallel universe, Man is just fine and living their days in the grey universe that we, through some as-yet-unphilosophized-but-don't-worry-we're-thinking-hard-about-it process, cannot exist in. ... but does that matter over-much if the observable is that ignoring zebra crossings gets you crossed out of this universe?
implements 2y ago
> Much of philosophy throughout history seems to operate this way. “Philosophy is poor at revealing truths but excellent at revealing falsehoods (or at least unsupported arguments)” was the main lesson I took from informally studying it.
yawboakye 2y ago
very few philosophers dared to live by their theories. the famous failures of aristotle (in the case of alexander) and plato in syracuse (where he saw firsthand that the philosopher-king is at best a book character) are good examples. seneca didn’t live stoically: he was avaricious and didn’t bother to incite civil war over unpaid debt, if ancient sources are to be believed. he failed horribly with nero, who later instructed him to commit suicide for treasonous crimes. again, if ancient sources are to be believed, he fumbled his suicide out of raw fear. the cynics, though, made a good life but that’s not because they had a better philosophy. it’s because cynicism is base/primitive logic available to the brute as well as the civilized man.
- lukan 2y ago
  And then there is Rousseau, who wrote books about raising children. But raising his own was too much.
  - yawboakye 2y ago
    rousseau is a fraud of the first order.
    - lukan 2y ago
      Enlightened fraud
  - shadowgovt 2y ago
    I actually have on my desk, right now, Mary Wollstonecraft's "A Vindication of the Rights of Woman." To summarize one of her points from memory, she basically lays into Rousseau about having a lot of opinions on domestic life for someone who clearly doesn't even know how to set foot in a kitchen.

waveBidder 2y ago

the term you're looking for is pascal's mugging, and it originates from within rationalism

IshKebab 2y ago
It seems a bit nonsense to me. > The mugger argues back that for any low but strictly greater than 0 probability of being able to pay back a large amount of money (or pure utility) there exists a finite amount that makes it rational to take the bet. This is a basic logic error. It ignores the very obvious fact that increasing the reward amount decreases the probability that it will be returned. E.g. if the probability of the reward R being returned is (0.5/R) we get "a low by strictly greater than 0 probability", and for that probability there is a (different) finite reward that would make it rational to take the bet, but it's not R. This is even simpler and more stupid than the "proofs" that 0=1. It does not change my opinion that philosophers (and lesswrong) are idiots.
- stuartjohnson12 2y ago
  That's what the person responding to meant - attempts to make human systems "rational" often involves simplifying dependent probabilities and presenting them as independent. The rationalist community both frequently makes this reasoning error and is aware they frequently make this error, and coined this term to refer to the category of reasoning mistake.
  - IshKebab 2y ago
    > coined this term to refer to the category of reasoning mistake. That's not at all what the Wikipedia article for it says. It presents it as an interesting paradox with several potential (and incorrect!) "remedies" rather than a category of basic logical errors.
    - close04 2y ago
      The “quadrillion days of happiness” offered to a rational person gives away that such allegories are anthropomorphized just for the sake of presentation. For the sake of what the philosophers mean, you should probably imagine this as an algorithm running on a machine (no AGI). It’s a mental tease, not a manual on how to react when faced with a mugger who forgot his weapon at home but has an interesting proposition to make. Similarly the trolley problem isn’t really about real humans in that situation, or else the correct answer would always be “do nothing”. It’s what the comment here [0] says. If you try to analyze everything purely rationally it will lead to dark corners of misunderstanding and madness. [0] https://news.ycombinator.com/item?id=42902931 https://news.ycombinator.com/item?id=42902931

michaelkeenan 2y ago

AGI would be extremely helpful in navigating clashes with aliens, but taking the time to make sure it's safe is very unlikely to make a difference to whether it's ready in time. Rationalists want AGI to be built, and they're generally very excited about it, e.g. many of them work at Anthropic. They just don't want a Move Fast and Break Things pace of development.

nkrisc 2y ago

It’s pointless. They aren’t rational. Any argument you come up with that contradicts their personal desires will be successfully “reasoned” away by them because they want it to be. Your mistake was ever thinking they had a rational thought to begin with, they think they are infallible.

HeatrayEnjoyer 2y ago
"Widespread robots that make their own decisions autonomously will probably be very bad for humans if they make decisions that aren't in our interest" isn't really that much of a stretch is it? If we were going slower maybe it would seem more theoretical. But there are multiple Manhattan-Project-level or (often, much) larger efforts ongoing to explicitly create software and robotic hardware that makes decisions and takes actions without any human in the loop. We don't need some kind of 10000 IQ god intelligence if a glitch token causes the majority of the labor force to suddenly and collectively engage in sabotage.
- NoMoreNicksLeft 2y ago
  None of those projects are even heading in the direction of "AGI". The state of AI today is something akin to what science fiction would have called an "oracle", a devise that can answer questions intelligently or seemingly-intelligently but otherwise isn't intelligent at all: it can't learn, has no agency, does nothing new. Even if that can be scaled up indefinitely, there's no reason to believe that it will ever become an AGI. If any of this can make decisions in a way that a human can, then I would start to question what human decision-making really amounts to.
  - rachofsunshine 2y ago
    For what it's worth, as the person at the top of this thread, I actually do take AI risk pretty seriously. Not in a singulatarian sense, but in the sense that I would be quite surprised if AI weren't capable of this stuff in ten years. Even the oracle version is already really dangerous in the wrong hands. A totalitarian government doesn't need to have someone listening to a few specific dissidents if they can have an AI transcriber write down every word of every phone conversation in the country, for example. And while it's certainly not error-proof, asking an LLM to do something like "go through this list of conversations and flag anything that sounds like anti-government sentiment" is going to get plenty of hits, too.

throwaway4aday 2y ago

It seems that you didn't understand the main point of the exposition. I'll summarize the ops comment a bit further. Points 1 and 2 only explain how they are able to erroneously justify their absurd beliefs, they don't explain why they hold those beliefs. Points 3 through 5 are the heart of the matter; egotistical and charismatic (to some types of people) leaders, open minded, freethinking and somewhat weird or marginalized people searching for meaning plus a way for them all to congregate around some shared interests. TLDR: perfect conditions for one or more cults to form.