7 ms·

> I've been saying this for years, and this is a large reason why I find the LessWrong folks to be almost entirely full of it. Their inability to come up with a

> I've been saying this for years, and this is a large reason why I find the LessWrong folks to be almost entirely full of it. Their inability to come up with accurate priors is completely lost on many of the folks who follow this kind of thinking. I assume you're saying LessWrong folks are more prone to miscalculating priors than most. Could you give some examples of this?

jacobolus 11y ago
The LessWrong folks aren’t obviously better or worse at calculating priors than anyone else. The “problem” is that their hobby is spending their free time considering outlandish scenarios, inventing arbitrary assumptions related to such scenarios, drawing questionable conclusions, and then convincing themselves that because they used logic and math, their analysis must be correct. Plenty of other folks who spend time on similar activities with a less pseudo-rigorous framing end up as conspiracy theorists or occultists; belief in AI overlords ruling humanity, the technological singularity, cryogenics, or impending 1000-year human lifespans is far from the kookiest thing people convince themselves about.
- TeMPOraL 11y ago
  Oh come on. That's how you're supposed to use math. To aid your thinking. Without it, and considering "outlandish scenarios", we would not have any scientific progress. Also, this is one strange thing - any time someone asks us (the STEM crowd), "what will I ever use math for in my life?", the default answer seems to be, "it's about having more tools for thinking, and greater clarity of thought; it'll make you smarter". But then, some of us turn around and refuse to acknowledge that people who actually learn math and try to apply it may be getting those promised results. Whether it's LW people, programmers, engineers or scientists, the moment it matters, the default conclusion is that math gives nothing.
  - jacobolus 11y ago
    What? Who said “math gives nothing”? I spend most of my day building things out of math. I think math and scientific inquiry are basically the most important tools invented/popularized in the past 1000 years. The lapse here is not math, but rather spending lots of attention on abstract thought disconnected from any kind of reality check. Of course, there’s nothing inherently wrong with speculating sans evidence about the future, it’s generally a harmless hobby. In the best case it makes for fun SF novels. Convincing yourself, still without direct evidence, that your speculation reflects truth implies that something has gone off the rails in the reasoning process, however. Using statistical analysis to understand real causal relationships in areas we have real data about is damn hard, and even plenty of people who are highly trained as statisticians screw up all the time. Academic fields like comparative politics (to take an example I spent a fair amount of time studying) are rife with poor conclusions drawn from bad analysis. The LessWrong folks are hardly unique in applying logic poorly. But they do tend to tackle more speculative questions and convince themselves more firmly of their conclusions (at least, such is my impression as an outsider).
    - Chathamization 11y ago
      I think this is a common problem when people working in fields that have somewhat accurate mathematical models look at fields that don't. They often don't realize how hard it is to create an accurate mathematical model for many situations, and assume that the other fields don't have them because the individuals who work in said fields aren't as good at math. Which is why every so often you'll get things like a physicist spending a couple months studying economics in their free time and deciding that they can now unlock the secret to economics which has eluded economists. This xkcd comic sums it up well: https://xkcd.com/793/ https://xkcd.com/793/
      - goatkcd 11y ago
        This goatkcd sums up that also http://goatkcd.com/793/sfw http://goatkcd.com/793/sfw
  - jacquesm 11y ago
    The more you extrapolate the more frequently you will need to adjust your future predictions because of errors in your initial measurements. This goes for any kind of extrapolation (for instance: plotting a course on a map), but it goes even more for extrapolating the future from limited evidence present today. Your 'best guess' might be off by many orders of magnitude if the evidence you have today is only loosely related to the future in terms of importance and where evidence may not be nearly as independent in nature as you currently perceive. This can lead to your best guess based on available evidence being about as good at predicting the future as randomness in spite of all the apparent effort at making the predictions mathematically sound.
- woodchuck64 11y ago
  But these aren't examples so much as vague caricatures. The subject matter that LessWrong considers is certainly unusual, but that alone should not be enough to call it arbitrary, questionable or outlandish.
  - Apocryphon 11y ago
    http://rationalwiki.org/wiki/Roko%27s_basilisk http://rationalwiki.org/wiki/Roko%27s_basilisk
    - unimpressive 11y ago
      https://en.wikipedia.org/wiki/Pascals_wager https://en.wikipedia.org/wiki/Pascals_wager I don't think it would be fair to malign philosophers because they come up with outlandish scary scenarios that scare people with OCD sometimes. It's not like LW gives Roko significant air time or serious treatment (EY freaking out and deleting it was partially principle of the thing, partially the fear that somebody might follow this road of thought to come up with something more terrifying and he doesn't want to take the community there, etc) somebody doing this is generally taken as a sign of serious crankery. (FWIW, I agree with the top parent post that the hyping of Bayes Theorem is one of the LW foibles. At least the presentation of it.)
      - eli_gottlieb 11y ago
        >I don't think it would be fair to malign philosophers because they come up with outlandish scary scenarios that scare people with OCD sometimes. Actually, that sounds like a fine reason to malign philosophers. In order to consider "outlandish scary scenarios", you must first be quite sure that those scenarios are realistic. If they're not, then you're wasting everyone's time. And yes, if your expected-utility expressions fail to converge because you believe in taking every Pascal's Wager/Mugging scenario into account, or because you don't believe in time, then you've attempted to take the limit of a nonconverging sequence and no amount of philosophizing will help.
      - Apocryphon 11y ago
- Anderkent 11y ago
  > because they used logic and math, their analysis must be correct > belief in AI overlords ruling humanity, the technological singularity, cryogenics, or impending 1000-year human lifespans I don't think anyone on LW believes these are 'facts' that are 'correct'. LW commonly thinks of these ideas as risks / opportunites that might happen (except for the 1000-year human lifespans, which is a new idea to me), and that it's probably worth investing minor amounts of money in case it does. In case of cryonics, that's about $20/month for insurance that covers it; in case of AI safety, that's a couple people doing research on the problem, and some amount of money sent their way. The way you phrase it seems like LW people are certain that cryonics will definitely let them be revived after death. That's definitely not the case - in fact, IIRC, on a survey a year or two ago LWers subscribed to cryonics assigned lower probability of it working than ones not subscribed. It's not a cargo-cult.
  - jacobolus 11y ago
    Substitute “plausible” for “correct” if you want to give them the benefit of the doubt. Either way it all so speculative as to be basically pure fiction. It reads very similar to me to various “scientific” defenses of particular religious traditions. Again, as I said, I don’t think there’s anything inherently wrong with this. Little communities of people should do whatever harmless hobby is fun for them. I just don’t find it very interesting or insightful. If people want to spend time and attention and resources on existential risks, how about the ones which are clear and imminent, like wealth inequality, the retreat of world democracy and increasing power of entirely unaccountable and amoral multinational corporations, or global climate change, e.g. http://www.esquire.com/news-politics/a36228/ballad-of-the-sad-climatologists-0815/ http://www.esquire.com/news-politics/a36228/ballad-of-the-sa...
- manish_gill 11y ago
  I've been saying t forever. Thanks for putting it so succinctly. LessWrong is a cult of people who want to be smart and they've essentially found a community in which certain assumptions and hypothetical scenarios combined with mathematical concepts make them think they've found the answer to everything in the Universe. They're no better than any other cult in my book. The problem is that it's only going to get worse with the advances in AI that are going on. Yudkowski has managed to convince some wealthy people to fund his so called research and we have OpenAI operating in the same waters which somehow gives LW people more legitimacy.
  - Anderkent 11y ago
    I don't understand how you can be so certain that the hypothetical scenarios they imagine can't possibly happen. Even if it really is laughable, we spend a lot of money on laughable research (homeopathy, anyone?), so why is this case so particularly bad?
    - manish_gill 11y ago
      ...Just because we've spent money on one laughable research doesn't mean we should spend it on more, does it? Are you seriously going to argue that?
      - Anderkent 11y ago
        No, what I'm arguing is that if we only spent money on things everyone agrees are useful, we'd never get anything done.
        manish_gill 11y ago
        Not everyone has to agree. That's why we have the scientific method and research institutes. If the current science shows that something is worth exploring in more detail, that some avenues are worth spending money on, spending money on them makes sense. Bullshit Ideas like AI apocalypse and singularity, transhumanism, downloading brain into a computer do not fit into that criteria.
- edanm 11y ago
  So I'm a LessWronger and know a bit about the "movement", and think you are misunderstanding what "LessWrongers think". Obviously not all LessWrongers think the same thing at all, but I'm talking about the average position of the people who believe AI safety should be worked towards. I'd love to explain the basic position, and tell me where you disagree with it. This is the basic position: 1. Intelligence can be created, because there is nothing "special"/"magical" about humans, and our intelligence was eventually created. 2. At some point, humanity will create an "artificial general intelligence". (Since we'll just keep improving science and technology, and there's no fundamental reason why this won't eventually allow us to create an intelligence. 3. "Artificial general intelligence" basically means a machine that is capable of achieving its goals, where the goals and methods it uses to achieve them are general. I.e. not "is able to play chess really well", but rather "is able to e.g. cure cancer". 4. For various reasons, once we have an artificial intelligence, it will likely become much smarter than us. (There are many reasons and debates about this, but let's just assume that since it's a computer, we can run it much faster than a human. If you dispute this point, we can talk about it more). 5. Something being much more intelligent than us means that, in effect, it has almost absolute power over what happens in the world (like we are basically all-powerful from the vantage point of monkeys, and their fate is totally in our hands). 6. (This is, I believe, the main point): Something being "intelligent" in the sense we're talking about doesn't say anything about what its goals are, or about how its mind works. We're used to everything that's intelligent being a human being, therefore the way our mind works is basically the same across every human. With an artificial intelligence, it will work completely differently from our mind. So if we "tell it" something like "cure cancer", it won't have our intuition and background knowledge to understand that we mean "but don't turn half the world into a giant computer in order to cure it". 7. Combine the two points above, and you get the large idea - whatever the goals of the AI will be, it will achieve them. Its goals won't, by default, be ones that are good for humanity, if only because we have no idea how to program our "value system" into a computer. 8. Therefore, we need to start working on making sure that when AI does come, it's safe. Even if we create an AI, the "extra" problem of making it safe is both hard, and we have absolutely no idea how to do it right now. We have no idea how long AI will take, or how long figuring out safety will take, but since this is a humanity-threatening problem, we should devote at least some resources to working on it right now. That's it, that's the basic idea. I'd love to hear which part you disagree with. I totally understand that not everyone will agree on some of the final details like, e.g., how many resources we should effectively devote right now (you might even claim it's 0 because anything we do now won't be useful). But I think the overall reasoning is sound, and would love to hear an intelligent disagreement.

jacobolus 11y ago

The LessWrong folks aren’t obviously better or worse at calculating priors than anyone else. The “problem” is that their hobby is spending their free time considering outlandish scenarios, inventing arbitrary assumptions related to such scenarios, drawing questionable conclusions, and then convincing themselves that because they used logic and math, their analysis must be correct. Plenty of other folks who spend time on similar activities with a less pseudo-rigorous framing end up as conspiracy theorists or occultists; belief in AI overlords ruling humanity, the technological singularity, cryogenics, or impending 1000-year human lifespans is far from the kookiest thing people convince themselves about.

TeMPOraL 11y ago
Oh come on. That's how you're supposed to use math. To aid your thinking. Without it, and considering "outlandish scenarios", we would not have any scientific progress. Also, this is one strange thing - any time someone asks us (the STEM crowd), "what will I ever use math for in my life?", the default answer seems to be, "it's about having more tools for thinking, and greater clarity of thought; it'll make you smarter". But then, some of us turn around and refuse to acknowledge that people who actually learn math and try to apply it may be getting those promised results. Whether it's LW people, programmers, engineers or scientists, the moment it matters, the default conclusion is that math gives nothing.
- jacobolus 11y ago
  What? Who said “math gives nothing”? I spend most of my day building things out of math. I think math and scientific inquiry are basically the most important tools invented/popularized in the past 1000 years. The lapse here is not math, but rather spending lots of attention on abstract thought disconnected from any kind of reality check. Of course, there’s nothing inherently wrong with speculating sans evidence about the future, it’s generally a harmless hobby. In the best case it makes for fun SF novels. Convincing yourself, still without direct evidence, that your speculation reflects truth implies that something has gone off the rails in the reasoning process, however. Using statistical analysis to understand real causal relationships in areas we have real data about is damn hard, and even plenty of people who are highly trained as statisticians screw up all the time. Academic fields like comparative politics (to take an example I spent a fair amount of time studying) are rife with poor conclusions drawn from bad analysis. The LessWrong folks are hardly unique in applying logic poorly. But they do tend to tackle more speculative questions and convince themselves more firmly of their conclusions (at least, such is my impression as an outsider).
  - Chathamization 11y ago
    I think this is a common problem when people working in fields that have somewhat accurate mathematical models look at fields that don't. They often don't realize how hard it is to create an accurate mathematical model for many situations, and assume that the other fields don't have them because the individuals who work in said fields aren't as good at math. Which is why every so often you'll get things like a physicist spending a couple months studying economics in their free time and deciding that they can now unlock the secret to economics which has eluded economists. This xkcd comic sums it up well: https://xkcd.com/793/ https://xkcd.com/793/
    - goatkcd 11y ago
      This goatkcd sums up that also http://goatkcd.com/793/sfw http://goatkcd.com/793/sfw
- jacquesm 11y ago
  The more you extrapolate the more frequently you will need to adjust your future predictions because of errors in your initial measurements. This goes for any kind of extrapolation (for instance: plotting a course on a map), but it goes even more for extrapolating the future from limited evidence present today. Your 'best guess' might be off by many orders of magnitude if the evidence you have today is only loosely related to the future in terms of importance and where evidence may not be nearly as independent in nature as you currently perceive. This can lead to your best guess based on available evidence being about as good at predicting the future as randomness in spite of all the apparent effort at making the predictions mathematically sound.
woodchuck64 11y ago
But these aren't examples so much as vague caricatures. The subject matter that LessWrong considers is certainly unusual, but that alone should not be enough to call it arbitrary, questionable or outlandish.
- Apocryphon 11y ago
  http://rationalwiki.org/wiki/Roko%27s_basilisk http://rationalwiki.org/wiki/Roko%27s_basilisk
  - unimpressive 11y ago
    https://en.wikipedia.org/wiki/Pascals_wager https://en.wikipedia.org/wiki/Pascals_wager I don't think it would be fair to malign philosophers because they come up with outlandish scary scenarios that scare people with OCD sometimes. It's not like LW gives Roko significant air time or serious treatment (EY freaking out and deleting it was partially principle of the thing, partially the fear that somebody might follow this road of thought to come up with something more terrifying and he doesn't want to take the community there, etc) somebody doing this is generally taken as a sign of serious crankery. (FWIW, I agree with the top parent post that the hyping of Bayes Theorem is one of the LW foibles. At least the presentation of it.)
    - eli_gottlieb 11y ago
      >I don't think it would be fair to malign philosophers because they come up with outlandish scary scenarios that scare people with OCD sometimes. Actually, that sounds like a fine reason to malign philosophers. In order to consider "outlandish scary scenarios", you must first be quite sure that those scenarios are realistic. If they're not, then you're wasting everyone's time. And yes, if your expected-utility expressions fail to converge because you believe in taking every Pascal's Wager/Mugging scenario into account, or because you don't believe in time, then you've attempted to take the limit of a nonconverging sequence and no amount of philosophizing will help.
    - Apocryphon 11y ago
Anderkent 11y ago
> because they used logic and math, their analysis must be correct > belief in AI overlords ruling humanity, the technological singularity, cryogenics, or impending 1000-year human lifespans I don't think anyone on LW believes these are 'facts' that are 'correct'. LW commonly thinks of these ideas as risks / opportunites that might happen (except for the 1000-year human lifespans, which is a new idea to me), and that it's probably worth investing minor amounts of money in case it does. In case of cryonics, that's about $20/month for insurance that covers it; in case of AI safety, that's a couple people doing research on the problem, and some amount of money sent their way. The way you phrase it seems like LW people are certain that cryonics will definitely let them be revived after death. That's definitely not the case - in fact, IIRC, on a survey a year or two ago LWers subscribed to cryonics assigned lower probability of it working than ones not subscribed. It's not a cargo-cult.
- jacobolus 11y ago
  Substitute “plausible” for “correct” if you want to give them the benefit of the doubt. Either way it all so speculative as to be basically pure fiction. It reads very similar to me to various “scientific” defenses of particular religious traditions. Again, as I said, I don’t think there’s anything inherently wrong with this. Little communities of people should do whatever harmless hobby is fun for them. I just don’t find it very interesting or insightful. If people want to spend time and attention and resources on existential risks, how about the ones which are clear and imminent, like wealth inequality, the retreat of world democracy and increasing power of entirely unaccountable and amoral multinational corporations, or global climate change, e.g. http://www.esquire.com/news-politics/a36228/ballad-of-the-sad-climatologists-0815/ http://www.esquire.com/news-politics/a36228/ballad-of-the-sa...
manish_gill 11y ago
I've been saying t forever. Thanks for putting it so succinctly. LessWrong is a cult of people who want to be smart and they've essentially found a community in which certain assumptions and hypothetical scenarios combined with mathematical concepts make them think they've found the answer to everything in the Universe. They're no better than any other cult in my book. The problem is that it's only going to get worse with the advances in AI that are going on. Yudkowski has managed to convince some wealthy people to fund his so called research and we have OpenAI operating in the same waters which somehow gives LW people more legitimacy.
- Anderkent 11y ago
  I don't understand how you can be so certain that the hypothetical scenarios they imagine can't possibly happen. Even if it really is laughable, we spend a lot of money on laughable research (homeopathy, anyone?), so why is this case so particularly bad?
  - manish_gill 11y ago
    ...Just because we've spent money on one laughable research doesn't mean we should spend it on more, does it? Are you seriously going to argue that?
    - Anderkent 11y ago
      No, what I'm arguing is that if we only spent money on things everyone agrees are useful, we'd never get anything done.
      - manish_gill 11y ago
        Not everyone has to agree. That's why we have the scientific method and research institutes. If the current science shows that something is worth exploring in more detail, that some avenues are worth spending money on, spending money on them makes sense. Bullshit Ideas like AI apocalypse and singularity, transhumanism, downloading brain into a computer do not fit into that criteria.
edanm 11y ago
So I'm a LessWronger and know a bit about the "movement", and think you are misunderstanding what "LessWrongers think". Obviously not all LessWrongers think the same thing at all, but I'm talking about the average position of the people who believe AI safety should be worked towards. I'd love to explain the basic position, and tell me where you disagree with it. This is the basic position: 1. Intelligence can be created, because there is nothing "special"/"magical" about humans, and our intelligence was eventually created. 2. At some point, humanity will create an "artificial general intelligence". (Since we'll just keep improving science and technology, and there's no fundamental reason why this won't eventually allow us to create an intelligence. 3. "Artificial general intelligence" basically means a machine that is capable of achieving its goals, where the goals and methods it uses to achieve them are general. I.e. not "is able to play chess really well", but rather "is able to e.g. cure cancer". 4. For various reasons, once we have an artificial intelligence, it will likely become much smarter than us. (There are many reasons and debates about this, but let's just assume that since it's a computer, we can run it much faster than a human. If you dispute this point, we can talk about it more). 5. Something being much more intelligent than us means that, in effect, it has almost absolute power over what happens in the world (like we are basically all-powerful from the vantage point of monkeys, and their fate is totally in our hands). 6. (This is, I believe, the main point): Something being "intelligent" in the sense we're talking about doesn't say anything about what its goals are, or about how its mind works. We're used to everything that's intelligent being a human being, therefore the way our mind works is basically the same across every human. With an artificial intelligence, it will work completely differently from our mind. So if we "tell it" something like "cure cancer", it won't have our intuition and background knowledge to understand that we mean "but don't turn half the world into a giant computer in order to cure it". 7. Combine the two points above, and you get the large idea - whatever the goals of the AI will be, it will achieve them. Its goals won't, by default, be ones that are good for humanity, if only because we have no idea how to program our "value system" into a computer. 8. Therefore, we need to start working on making sure that when AI does come, it's safe. Even if we create an AI, the "extra" problem of making it safe is both hard, and we have absolutely no idea how to do it right now. We have no idea how long AI will take, or how long figuring out safety will take, but since this is a humanity-threatening problem, we should devote at least some resources to working on it right now. That's it, that's the basic idea. I'd love to hear which part you disagree with. I totally understand that not everyone will agree on some of the final details like, e.g., how many resources we should effectively devote right now (you might even claim it's 0 because anything we do now won't be useful). But I think the overall reasoning is sound, and would love to hear an intelligent disagreement.