25 ms·

I've been saying this for years, and this is a large reason why I find the LessWrong folks to be almost entirely full of it. Their inability to come up with ac

by Laaw 11y ago

I've been saying this for years, and this is a large reason why I find the LessWrong folks to be almost entirely full of it. Their inability to come up with accurate priors is completely lost on many of the folks who follow this kind of thinking. A couple of comments are saying, "no duh" to this article, but those folks likely don't realize quite how many other people are falling into this trap. "Garbage in, garbage out" is only good advice when the person you're saying it to realizes they're putting garbage in.

woodchuck64 11y ago
> I've been saying this for years, and this is a large reason why I find the LessWrong folks to be almost entirely full of it. Their inability to come up with accurate priors is completely lost on many of the folks who follow this kind of thinking. I assume you're saying LessWrong folks are more prone to miscalculating priors than most. Could you give some examples of this?
- jacobolus 11y ago
  The LessWrong folks aren’t obviously better or worse at calculating priors than anyone else. The “problem” is that their hobby is spending their free time considering outlandish scenarios, inventing arbitrary assumptions related to such scenarios, drawing questionable conclusions, and then convincing themselves that because they used logic and math, their analysis must be correct. Plenty of other folks who spend time on similar activities with a less pseudo-rigorous framing end up as conspiracy theorists or occultists; belief in AI overlords ruling humanity, the technological singularity, cryogenics, or impending 1000-year human lifespans is far from the kookiest thing people convince themselves about.
  - TeMPOraL 11y ago
    Oh come on. That's how you're supposed to use math. To aid your thinking. Without it, and considering "outlandish scenarios", we would not have any scientific progress. Also, this is one strange thing - any time someone asks us (the STEM crowd), "what will I ever use math for in my life?", the default answer seems to be, "it's about having more tools for thinking, and greater clarity of thought; it'll make you smarter". But then, some of us turn around and refuse to acknowledge that people who actually learn math and try to apply it may be getting those promised results. Whether it's LW people, programmers, engineers or scientists, the moment it matters, the default conclusion is that math gives nothing.
    - jacobolus 11y ago
      What? Who said “math gives nothing”? I spend most of my day building things out of math. I think math and scientific inquiry are basically the most important tools invented/popularized in the past 1000 years. The lapse here is not math, but rather spending lots of attention on abstract thought disconnected from any kind of reality check. Of course, there’s nothing inherently wrong with speculating sans evidence about the future, it’s generally a harmless hobby. In the best case it makes for fun SF novels. Convincing yourself, still without direct evidence, that your speculation reflects truth implies that something has gone off the rails in the reasoning process, however. Using statistical analysis to understand real causal relationships in areas we have real data about is damn hard, and even plenty of people who are highly trained as statisticians screw up all the time. Academic fields like comparative politics (to take an example I spent a fair amount of time studying) are rife with poor conclusions drawn from bad analysis. The LessWrong folks are hardly unique in applying logic poorly. But they do tend to tackle more speculative questions and convince themselves more firmly of their conclusions (at least, such is my impression as an outsider).
      - Chathamization 11y ago
        I think this is a common problem when people working in fields that have somewhat accurate mathematical models look at fields that don't. They often don't realize how hard it is to create an accurate mathematical model for many situations, and assume that the other fields don't have them because the individuals who work in said fields aren't as good at math. Which is why every so often you'll get things like a physicist spending a couple months studying economics in their free time and deciding that they can now unlock the secret to economics which has eluded economists. This xkcd comic sums it up well: https://xkcd.com/793/ https://xkcd.com/793/
        goatkcd 11y ago
        This goatkcd sums up that also http://goatkcd.com/793/sfw http://goatkcd.com/793/sfw
    - jacquesm 11y ago
  - woodchuck64 11y ago
    But these aren't examples so much as vague caricatures. The subject matter that LessWrong considers is certainly unusual, but that alone should not be enough to call it arbitrary, questionable or outlandish.
    - Apocryphon 11y ago
      http://rationalwiki.org/wiki/Roko%27s_basilisk http://rationalwiki.org/wiki/Roko%27s_basilisk
      - unimpressive 11y ago
        https://en.wikipedia.org/wiki/Pascals_wager https://en.wikipedia.org/wiki/Pascals_wager I don't think it would be fair to malign philosophers because they come up with outlandish scary scenarios that scare people with OCD sometimes. It's not like LW gives Roko significant air time or serious treatment (EY freaking out and deleting it was partially principle of the thing, partially the fear that somebody might follow this road of thought to come up with something more terrifying and he doesn't want to take the community there, etc) somebody doing this is generally taken as a sign of serious crankery. (FWIW, I agree with the top parent post that the hyping of Bayes Theorem is one of the LW foibles. At least the presentation of it.)
        eli_gottlieb 11y ago
        >I don't think it would be fair to malign philosophers because they come up with outlandish scary scenarios that scare people with OCD sometimes. Actually, that sounds like a fine reason to malign philosophers. In order to consider "outlandish scary scenarios", you must first be quite sure that those scenarios are realistic. If they're not, then you're wasting everyone's time. And yes, if your expected-utility expressions fail to converge because you believe in taking every Pascal's Wager/Mugging scenario into account, or because you don't believe in time, then you've attempted to take the limit of a nonconverging sequence and no amount of philosophizing will help.
        Apocryphon 11y ago
  - Anderkent 11y ago
    > because they used logic and math, their analysis must be correct > belief in AI overlords ruling humanity, the technological singularity, cryogenics, or impending 1000-year human lifespans I don't think anyone on LW believes these are 'facts' that are 'correct'. LW commonly thinks of these ideas as risks / opportunites that might happen (except for the 1000-year human lifespans, which is a new idea to me), and that it's probably worth investing minor amounts of money in case it does. In case of cryonics, that's about $20/month for insurance that covers it; in case of AI safety, that's a couple people doing research on the problem, and some amount of money sent their way. The way you phrase it seems like LW people are certain that cryonics will definitely let them be revived after death. That's definitely not the case - in fact, IIRC, on a survey a year or two ago LWers subscribed to cryonics assigned lower probability of it working than ones not subscribed. It's not a cargo-cult.
    - jacobolus 11y ago
      Substitute “plausible” for “correct” if you want to give them the benefit of the doubt. Either way it all so speculative as to be basically pure fiction. It reads very similar to me to various “scientific” defenses of particular religious traditions. Again, as I said, I don’t think there’s anything inherently wrong with this. Little communities of people should do whatever harmless hobby is fun for them. I just don’t find it very interesting or insightful. If people want to spend time and attention and resources on existential risks, how about the ones which are clear and imminent, like wealth inequality, the retreat of world democracy and increasing power of entirely unaccountable and amoral multinational corporations, or global climate change, e.g. http://www.esquire.com/news-politics/a36228/ballad-of-the-sad-climatologists-0815/ http://www.esquire.com/news-politics/a36228/ballad-of-the-sa...
  - manish_gill 11y ago
    I've been saying t forever. Thanks for putting it so succinctly. LessWrong is a cult of people who want to be smart and they've essentially found a community in which certain assumptions and hypothetical scenarios combined with mathematical concepts make them think they've found the answer to everything in the Universe. They're no better than any other cult in my book. The problem is that it's only going to get worse with the advances in AI that are going on. Yudkowski has managed to convince some wealthy people to fund his so called research and we have OpenAI operating in the same waters which somehow gives LW people more legitimacy.
    - Anderkent 11y ago
      I don't understand how you can be so certain that the hypothetical scenarios they imagine can't possibly happen. Even if it really is laughable, we spend a lot of money on laughable research (homeopathy, anyone?), so why is this case so particularly bad?
      - manish_gill 11y ago
        ...Just because we've spent money on one laughable research doesn't mean we should spend it on more, does it? Are you seriously going to argue that?
        Anderkent 11y ago
        No, what I'm arguing is that if we only spent money on things everyone agrees are useful, we'd never get anything done.
        manish_gill 11y ago
        Not everyone has to agree. That's why we have the scientific method and research institutes. If the current science shows that something is worth exploring in more detail, that some avenues are worth spending money on, spending money on them makes sense. Bullshit Ideas like AI apocalypse and singularity, transhumanism, downloading brain into a computer do not fit into that criteria.
  - edanm 11y ago
    So I'm a LessWronger and know a bit about the "movement", and think you are misunderstanding what "LessWrongers think". Obviously not all LessWrongers think the same thing at all, but I'm talking about the average position of the people who believe AI safety should be worked towards. I'd love to explain the basic position, and tell me where you disagree with it. This is the basic position: 1. Intelligence can be created, because there is nothing "special"/"magical" about humans, and our intelligence was eventually created. 2. At some point, humanity will create an "artificial general intelligence". (Since we'll just keep improving science and technology, and there's no fundamental reason why this won't eventually allow us to create an intelligence. 3. "Artificial general intelligence" basically means a machine that is capable of achieving its goals, where the goals and methods it uses to achieve them are general. I.e. not "is able to play chess really well", but rather "is able to e.g. cure cancer". 4. For various reasons, once we have an artificial intelligence, it will likely become much smarter than us. (There are many reasons and debates about this, but let's just assume that since it's a computer, we can run it much faster than a human. If you dispute this point, we can talk about it more). 5. Something being much more intelligent than us means that, in effect, it has almost absolute power over what happens in the world (like we are basically all-powerful from the vantage point of monkeys, and their fate is totally in our hands). 6. (This is, I believe, the main point): Something being "intelligent" in the sense we're talking about doesn't say anything about what its goals are, or about how its mind works. We're used to everything that's intelligent being a human being, therefore the way our mind works is basically the same across every human. With an artificial intelligence, it will work completely differently from our mind. So if we "tell it" something like "cure cancer", it won't have our intuition and background knowledge to understand that we mean "but don't turn half the world into a giant computer in order to cure it". 7. Combine the two points above, and you get the large idea - whatever the goals of the AI will be, it will achieve them. Its goals won't, by default, be ones that are good for humanity, if only because we have no idea how to program our "value system" into a computer. 8. Therefore, we need to start working on making sure that when AI does come, it's safe. Even if we create an AI, the "extra" problem of making it safe is both hard, and we have absolutely no idea how to do it right now. We have no idea how long AI will take, or how long figuring out safety will take, but since this is a humanity-threatening problem, we should devote at least some resources to working on it right now. That's it, that's the basic idea. I'd love to hear which part you disagree with. I totally understand that not everyone will agree on some of the final details like, e.g., how many resources we should effectively devote right now (you might even claim it's 0 because anything we do now won't be useful). But I think the overall reasoning is sound, and would love to hear an intelligent disagreement.
IanCal 11y ago
Do priors just start you off closer to the truth? That is to say, if you start with any prior, will enough additional pieces of evidence always let you converge on the truth? Does anyone commonly set their priors to be a distribution? Perhaps a range or actually a normal distribution to represent a prior with uncertainty?
- XFrequentist 11y ago
  Yes and yes and yes! ETA - caveat to the 2nd question: ... unless you've restricted your prior to exclude the truth.
- Fomite 11y ago
  In my field (Epidemiology), when doing Bayesian analysis, it is very common to set one's priors to be a distribution. Sometimes the point estimate and spread of a previously conducted study or meta-analysis, sometimes merely a uniform distribution with upper and lower bounds ("It is extremely unlikely that the relative risk of disease for this exposure is below 0.01 or above 100...") It's been argued that frequentist analysis is essentially a Bayesian analysis with a prior distribution centered on zero with bounds from positive to negative infinity.
  - skybrian 11y ago
    How does that work? It doesn't sound like a well-defined distribution since the area under the curve needs to be 1.
    - Fomite 11y ago
      It's more of a philosophical statement than an actual implementation. Mainly that frequentist analysis begins with the prior "I dunno, could be anything..." See also: The calculus answer.
    - bmh100 11y ago
      It has to do with calculus. If the probability of a given result approaches zero as the result itself approaches positive or negative infinity, then the area under the curve approaches 1. Imagine 1/2 + 1/4 + 1/8 ... to infinity. The sum approaches 1 as the denominator approaches infinity. With calculus, we can determine that with mathematical methods.
XFrequentist 11y ago
FWIW, it seems to me that a major benefit of the Bayesian approach is to make bad reasoning (in the form of, say, an unreasonable prior) transparent and obvious. I've never heard it claimed that the Bayesian approach was robust to sophisticated idiocy (neither on LessWrong nor mainstream writing on Bayesian methods), except in the narrow techical sense that the posterior asymptotically approximates the likelihood given infinite data (provably true under some assumptions but irrelevant to the objections in the OP).
- TeMPOraL 11y ago
  Yup. Actually, a common theme on LessWrong was realizing that with better reasoning tools you're more able to bullshit yourself, and so you need to be extra-careful.
  - zepto 11y ago
    How do you be extra careful except by developing yet more powerful reasoning tools?
    - TeMPOraL 11y ago
      You can't protect yourself in 100% - it would require developing more powerful reasoning tools in an infinite regression. But what you can do is to use introspection, and triple-check your reasoning when it seems to defy common sense or leads you to weird (awful) conclusions. That's why LW is so big on biases and heuristics by the way - you can treat them as a list of warning signs; if your reasoning seems to match some of them, it's time to take a closer look. (A thing a lot of people missed since knowing logical fallacies started to become a mainstream thing - you should use the list to check your own reasoning, not your opponent's.)
      - Chathamization 11y ago
        The problem with trying to rely on heuristics to avoid biases is people often ignore the biases in the heuristics of choice. To continue the example of LW, there are many people there who seem to think highly of IQ test, and who ignore the many issues with them (the Flynn effect an the effect of incentives being a couple examples of the flaws in IQ tests). Trying to remove biases is great. But there is a problem when someone works to remove some biases, then believes that they are inherently more rational than the public at large, and then uncritically accepts there other biases ("Someone like me who's worked hard to remove there biases must be correct when compared to the biased masses.").
arcanus 11y ago
> Their inability to come up with accurate priors is completely lost on many of the folks who follow this kind of thinking. Mis-specified/faulty priors are absolutely a possible pitfall of Bayes, but Jeffreys Priors are a means to circumvent this: https://en.wikipedia.org/wiki/Jeffreys_prior https://en.wikipedia.org/wiki/Jeffreys_prior Another thing not mentioned in the article is that Bayes DOES converge to the frequentist result in the limit of big data.
BenoitEssiambre 11y ago
Except that at least with Bayesian methods the prior is explicitly laid out. Frequencist methods when they are used to make predictions and get useful information out of experiments have hidden implicit priors that bias inferences in opaque ways. The very honest frequencists will admit that their procedures are only rejecting hypotheses so small as to have no practical utility. Others will use weird and dishonest doublespeak where they call rejecting an insignificantly small hypothesis "statistical significance". But I suppose they do redeem themselves a bit with the wording "null hypothesis" which candidly conveys the sense of having significantly rejected _nothing_.
- GFK_of_xmaspast 11y ago
  If frequentism is so terrible, why has the field of statistics had so much success over the last century or so.
  - btown 11y ago
    Just because something is better than nothing, doesn't mean it's optimal.
mjgeddes 11y ago
Bayesianism is a 'grand unified theory of reasoning' that all of science be should be based on assigning (and updating) probabilities for a list of possible outcomes; the probabilities are supposed to indicate your subjective degree of confidence that a given outcome will occur. Contrast this with an alternative conception of rationality as espoused by David Deutsch. David Deutsch in his superb books, 'The Fabric Of Theory' and 'The Beginning Of Infinity', argued for a different theory of reasoning than Bayesianism. Deutsch (correctly in my view) pointed out that real science is not based on probabilistic predictions, but on explanations. So real science is better thought of as the growth or integration of knowledge, rather than probability calculations. So what's wrong with Bayesianism? Probability theory was designed for reasoning about external observations - sensory data. (for example, "a coin has a 50% chance of coming up heads"). In terms of predicting things in the external world, it works very well. Where it breaks down is when you try to apply it to reasoning about your own internal thought processes. It was never intended to do this. As statistician Andrew Gelman correctly points out, it is simply invalid to try to assign probabilities to mathematical statements or theories, for instance. Can an alternative mathematical framework be developed, one more in keeping with the ideas of David Deutsch and the coherence theory of knowledge? I believe the answer is yes, and I am going to sketch the basic ideas for such a framework. The basic idea is to separate out levels of abstraction when reasoning (or equivalently, levels of recursion). In my proposed framework, there are 3 levels, and each level gets its own measure of 'truth-value'. All reasoning must terminate in a Boolean truth value (True/False) at the base level but the idea is that different forms of reasoning correspond to different levels of abstraction. 1st level: Boolean logic (True/False) 2nd level: Probability value (0-1) 3rd level: Conceptual coherence (categorization measure) For full reflection, you need three different numbers: a Boolean value (T/F) at the base, a probability value (0-1) at the next level of abstraction, and an entirely new measure called conceptual coherence at the highest of abstraction. As a rough working definition of conceptual coherence, I would define it thusly; "The degree to which a concept coheres with (integrates with) the overall world-model." It should now be clear what's wrong with Bayesianism! It only gets us to the 2nd level abstraction! There is not just uncertainty about our own knowledge of the world (probability), there is another meta-level of uncertainly; uncertainty about our own reasoning processes, or logical uncertainty. Bayesianism can't help us here. Conceptual coherence can. Lets see how: All statements of the form: ‘outcome x has probability y’ can be converted into statements about conceptual coherence, simply by redefining ‘x’ as a concept in a world-model. Then the correct form of logical expression is: ‘concept x has coherence value y’. The idea is that probability values are just special cases of coherence (the notion of coherence is more general than the notion of probabilities). To conclude, conceptual coherence is the degree with which a concept is integrated with the rest of your world-model, and I think it accurately captures in mathematical terms the ideas that Deutsch was trying express, and is a more powerful method of reasoning than Bayesianism.

by Laaw 11y ago

woodchuck64 11y ago

> I've been saying this for years, and this is a large reason why I find the LessWrong folks to be almost entirely full of it. Their inability to come up with accurate priors is completely lost on many of the folks who follow this kind of thinking. I assume you're saying LessWrong folks are more prone to miscalculating priors than most. Could you give some examples of this?

jacobolus 11y ago
The LessWrong folks aren’t obviously better or worse at calculating priors than anyone else. The “problem” is that their hobby is spending their free time considering outlandish scenarios, inventing arbitrary assumptions related to such scenarios, drawing questionable conclusions, and then convincing themselves that because they used logic and math, their analysis must be correct. Plenty of other folks who spend time on similar activities with a less pseudo-rigorous framing end up as conspiracy theorists or occultists; belief in AI overlords ruling humanity, the technological singularity, cryogenics, or impending 1000-year human lifespans is far from the kookiest thing people convince themselves about.
- TeMPOraL 11y ago
  Oh come on. That's how you're supposed to use math. To aid your thinking. Without it, and considering "outlandish scenarios", we would not have any scientific progress. Also, this is one strange thing - any time someone asks us (the STEM crowd), "what will I ever use math for in my life?", the default answer seems to be, "it's about having more tools for thinking, and greater clarity of thought; it'll make you smarter". But then, some of us turn around and refuse to acknowledge that people who actually learn math and try to apply it may be getting those promised results. Whether it's LW people, programmers, engineers or scientists, the moment it matters, the default conclusion is that math gives nothing.
  - jacobolus 11y ago
    What? Who said “math gives nothing”? I spend most of my day building things out of math. I think math and scientific inquiry are basically the most important tools invented/popularized in the past 1000 years. The lapse here is not math, but rather spending lots of attention on abstract thought disconnected from any kind of reality check. Of course, there’s nothing inherently wrong with speculating sans evidence about the future, it’s generally a harmless hobby. In the best case it makes for fun SF novels. Convincing yourself, still without direct evidence, that your speculation reflects truth implies that something has gone off the rails in the reasoning process, however. Using statistical analysis to understand real causal relationships in areas we have real data about is damn hard, and even plenty of people who are highly trained as statisticians screw up all the time. Academic fields like comparative politics (to take an example I spent a fair amount of time studying) are rife with poor conclusions drawn from bad analysis. The LessWrong folks are hardly unique in applying logic poorly. But they do tend to tackle more speculative questions and convince themselves more firmly of their conclusions (at least, such is my impression as an outsider).
    - Chathamization 11y ago
      I think this is a common problem when people working in fields that have somewhat accurate mathematical models look at fields that don't. They often don't realize how hard it is to create an accurate mathematical model for many situations, and assume that the other fields don't have them because the individuals who work in said fields aren't as good at math. Which is why every so often you'll get things like a physicist spending a couple months studying economics in their free time and deciding that they can now unlock the secret to economics which has eluded economists. This xkcd comic sums it up well: https://xkcd.com/793/ https://xkcd.com/793/
      - goatkcd 11y ago
        This goatkcd sums up that also http://goatkcd.com/793/sfw http://goatkcd.com/793/sfw
  - jacquesm 11y ago
- woodchuck64 11y ago
  But these aren't examples so much as vague caricatures. The subject matter that LessWrong considers is certainly unusual, but that alone should not be enough to call it arbitrary, questionable or outlandish.
  - Apocryphon 11y ago
    http://rationalwiki.org/wiki/Roko%27s_basilisk http://rationalwiki.org/wiki/Roko%27s_basilisk
    - unimpressive 11y ago
      https://en.wikipedia.org/wiki/Pascals_wager https://en.wikipedia.org/wiki/Pascals_wager I don't think it would be fair to malign philosophers because they come up with outlandish scary scenarios that scare people with OCD sometimes. It's not like LW gives Roko significant air time or serious treatment (EY freaking out and deleting it was partially principle of the thing, partially the fear that somebody might follow this road of thought to come up with something more terrifying and he doesn't want to take the community there, etc) somebody doing this is generally taken as a sign of serious crankery. (FWIW, I agree with the top parent post that the hyping of Bayes Theorem is one of the LW foibles. At least the presentation of it.)
      - eli_gottlieb 11y ago
        >I don't think it would be fair to malign philosophers because they come up with outlandish scary scenarios that scare people with OCD sometimes. Actually, that sounds like a fine reason to malign philosophers. In order to consider "outlandish scary scenarios", you must first be quite sure that those scenarios are realistic. If they're not, then you're wasting everyone's time. And yes, if your expected-utility expressions fail to converge because you believe in taking every Pascal's Wager/Mugging scenario into account, or because you don't believe in time, then you've attempted to take the limit of a nonconverging sequence and no amount of philosophizing will help.
      - Apocryphon 11y ago
- Anderkent 11y ago
  > because they used logic and math, their analysis must be correct > belief in AI overlords ruling humanity, the technological singularity, cryogenics, or impending 1000-year human lifespans I don't think anyone on LW believes these are 'facts' that are 'correct'. LW commonly thinks of these ideas as risks / opportunites that might happen (except for the 1000-year human lifespans, which is a new idea to me), and that it's probably worth investing minor amounts of money in case it does. In case of cryonics, that's about $20/month for insurance that covers it; in case of AI safety, that's a couple people doing research on the problem, and some amount of money sent their way. The way you phrase it seems like LW people are certain that cryonics will definitely let them be revived after death. That's definitely not the case - in fact, IIRC, on a survey a year or two ago LWers subscribed to cryonics assigned lower probability of it working than ones not subscribed. It's not a cargo-cult.
  - jacobolus 11y ago
    Substitute “plausible” for “correct” if you want to give them the benefit of the doubt. Either way it all so speculative as to be basically pure fiction. It reads very similar to me to various “scientific” defenses of particular religious traditions. Again, as I said, I don’t think there’s anything inherently wrong with this. Little communities of people should do whatever harmless hobby is fun for them. I just don’t find it very interesting or insightful. If people want to spend time and attention and resources on existential risks, how about the ones which are clear and imminent, like wealth inequality, the retreat of world democracy and increasing power of entirely unaccountable and amoral multinational corporations, or global climate change, e.g. http://www.esquire.com/news-politics/a36228/ballad-of-the-sad-climatologists-0815/ http://www.esquire.com/news-politics/a36228/ballad-of-the-sa...
- manish_gill 11y ago
  I've been saying t forever. Thanks for putting it so succinctly. LessWrong is a cult of people who want to be smart and they've essentially found a community in which certain assumptions and hypothetical scenarios combined with mathematical concepts make them think they've found the answer to everything in the Universe. They're no better than any other cult in my book. The problem is that it's only going to get worse with the advances in AI that are going on. Yudkowski has managed to convince some wealthy people to fund his so called research and we have OpenAI operating in the same waters which somehow gives LW people more legitimacy.
  - Anderkent 11y ago
    I don't understand how you can be so certain that the hypothetical scenarios they imagine can't possibly happen. Even if it really is laughable, we spend a lot of money on laughable research (homeopathy, anyone?), so why is this case so particularly bad?
    - manish_gill 11y ago
      ...Just because we've spent money on one laughable research doesn't mean we should spend it on more, does it? Are you seriously going to argue that?
      - Anderkent 11y ago
        No, what I'm arguing is that if we only spent money on things everyone agrees are useful, we'd never get anything done.
        manish_gill 11y ago
        Not everyone has to agree. That's why we have the scientific method and research institutes. If the current science shows that something is worth exploring in more detail, that some avenues are worth spending money on, spending money on them makes sense. Bullshit Ideas like AI apocalypse and singularity, transhumanism, downloading brain into a computer do not fit into that criteria.
- edanm 11y ago
  So I'm a LessWronger and know a bit about the "movement", and think you are misunderstanding what "LessWrongers think". Obviously not all LessWrongers think the same thing at all, but I'm talking about the average position of the people who believe AI safety should be worked towards. I'd love to explain the basic position, and tell me where you disagree with it. This is the basic position: 1. Intelligence can be created, because there is nothing "special"/"magical" about humans, and our intelligence was eventually created. 2. At some point, humanity will create an "artificial general intelligence". (Since we'll just keep improving science and technology, and there's no fundamental reason why this won't eventually allow us to create an intelligence. 3. "Artificial general intelligence" basically means a machine that is capable of achieving its goals, where the goals and methods it uses to achieve them are general. I.e. not "is able to play chess really well", but rather "is able to e.g. cure cancer". 4. For various reasons, once we have an artificial intelligence, it will likely become much smarter than us. (There are many reasons and debates about this, but let's just assume that since it's a computer, we can run it much faster than a human. If you dispute this point, we can talk about it more). 5. Something being much more intelligent than us means that, in effect, it has almost absolute power over what happens in the world (like we are basically all-powerful from the vantage point of monkeys, and their fate is totally in our hands). 6. (This is, I believe, the main point): Something being "intelligent" in the sense we're talking about doesn't say anything about what its goals are, or about how its mind works. We're used to everything that's intelligent being a human being, therefore the way our mind works is basically the same across every human. With an artificial intelligence, it will work completely differently from our mind. So if we "tell it" something like "cure cancer", it won't have our intuition and background knowledge to understand that we mean "but don't turn half the world into a giant computer in order to cure it". 7. Combine the two points above, and you get the large idea - whatever the goals of the AI will be, it will achieve them. Its goals won't, by default, be ones that are good for humanity, if only because we have no idea how to program our "value system" into a computer. 8. Therefore, we need to start working on making sure that when AI does come, it's safe. Even if we create an AI, the "extra" problem of making it safe is both hard, and we have absolutely no idea how to do it right now. We have no idea how long AI will take, or how long figuring out safety will take, but since this is a humanity-threatening problem, we should devote at least some resources to working on it right now. That's it, that's the basic idea. I'd love to hear which part you disagree with. I totally understand that not everyone will agree on some of the final details like, e.g., how many resources we should effectively devote right now (you might even claim it's 0 because anything we do now won't be useful). But I think the overall reasoning is sound, and would love to hear an intelligent disagreement.

IanCal 11y ago

Do priors just start you off closer to the truth? That is to say, if you start with any prior, will enough additional pieces of evidence always let you converge on the truth? Does anyone commonly set their priors to be a distribution? Perhaps a range or actually a normal distribution to represent a prior with uncertainty?

XFrequentist 11y ago
Yes and yes and yes! ETA - caveat to the 2nd question: ... unless you've restricted your prior to exclude the truth.
Fomite 11y ago
In my field (Epidemiology), when doing Bayesian analysis, it is very common to set one's priors to be a distribution. Sometimes the point estimate and spread of a previously conducted study or meta-analysis, sometimes merely a uniform distribution with upper and lower bounds ("It is extremely unlikely that the relative risk of disease for this exposure is below 0.01 or above 100...") It's been argued that frequentist analysis is essentially a Bayesian analysis with a prior distribution centered on zero with bounds from positive to negative infinity.
- skybrian 11y ago
  How does that work? It doesn't sound like a well-defined distribution since the area under the curve needs to be 1.
  - Fomite 11y ago
    It's more of a philosophical statement than an actual implementation. Mainly that frequentist analysis begins with the prior "I dunno, could be anything..." See also: The calculus answer.
  - bmh100 11y ago
    It has to do with calculus. If the probability of a given result approaches zero as the result itself approaches positive or negative infinity, then the area under the curve approaches 1. Imagine 1/2 + 1/4 + 1/8 ... to infinity. The sum approaches 1 as the denominator approaches infinity. With calculus, we can determine that with mathematical methods.

XFrequentist 11y ago

FWIW, it seems to me that a major benefit of the Bayesian approach is to make bad reasoning (in the form of, say, an unreasonable prior) transparent and obvious. I've never heard it claimed that the Bayesian approach was robust to sophisticated idiocy (neither on LessWrong nor mainstream writing on Bayesian methods), except in the narrow techical sense that the posterior asymptotically approximates the likelihood given infinite data (provably true under some assumptions but irrelevant to the objections in the OP).

TeMPOraL 11y ago
Yup. Actually, a common theme on LessWrong was realizing that with better reasoning tools you're more able to bullshit yourself, and so you need to be extra-careful.
- zepto 11y ago
  How do you be extra careful except by developing yet more powerful reasoning tools?
  - TeMPOraL 11y ago
    You can't protect yourself in 100% - it would require developing more powerful reasoning tools in an infinite regression. But what you can do is to use introspection, and triple-check your reasoning when it seems to defy common sense or leads you to weird (awful) conclusions. That's why LW is so big on biases and heuristics by the way - you can treat them as a list of warning signs; if your reasoning seems to match some of them, it's time to take a closer look. (A thing a lot of people missed since knowing logical fallacies started to become a mainstream thing - you should use the list to check your own reasoning, not your opponent's.)
    - Chathamization 11y ago
      The problem with trying to rely on heuristics to avoid biases is people often ignore the biases in the heuristics of choice. To continue the example of LW, there are many people there who seem to think highly of IQ test, and who ignore the many issues with them (the Flynn effect an the effect of incentives being a couple examples of the flaws in IQ tests). Trying to remove biases is great. But there is a problem when someone works to remove some biases, then believes that they are inherently more rational than the public at large, and then uncritically accepts there other biases ("Someone like me who's worked hard to remove there biases must be correct when compared to the biased masses.").

arcanus 11y ago

> Their inability to come up with accurate priors is completely lost on many of the folks who follow this kind of thinking. Mis-specified/faulty priors are absolutely a possible pitfall of Bayes, but Jeffreys Priors are a means to circumvent this: https://en.wikipedia.org/wiki/Jeffreys_prior https://en.wikipedia.org/wiki/Jeffreys_prior Another thing not mentioned in the article is that Bayes DOES converge to the frequentist result in the limit of big data.

BenoitEssiambre 11y ago

Except that at least with Bayesian methods the prior is explicitly laid out. Frequencist methods when they are used to make predictions and get useful information out of experiments have hidden implicit priors that bias inferences in opaque ways. The very honest frequencists will admit that their procedures are only rejecting hypotheses so small as to have no practical utility. Others will use weird and dishonest doublespeak where they call rejecting an insignificantly small hypothesis "statistical significance". But I suppose they do redeem themselves a bit with the wording "null hypothesis" which candidly conveys the sense of having significantly rejected _nothing_.

GFK_of_xmaspast 11y ago
If frequentism is so terrible, why has the field of statistics had so much success over the last century or so.
- btown 11y ago
  Just because something is better than nothing, doesn't mean it's optimal.

mjgeddes 11y ago

Bayesianism is a 'grand unified theory of reasoning' that all of science be should be based on assigning (and updating) probabilities for a list of possible outcomes; the probabilities are supposed to indicate your subjective degree of confidence that a given outcome will occur. Contrast this with an alternative conception of rationality as espoused by David Deutsch. David Deutsch in his superb books, 'The Fabric Of Theory' and 'The Beginning Of Infinity', argued for a different theory of reasoning than Bayesianism. Deutsch (correctly in my view) pointed out that real science is not based on probabilistic predictions, but on explanations. So real science is better thought of as the growth or integration of knowledge, rather than probability calculations. So what's wrong with Bayesianism? Probability theory was designed for reasoning about external observations - sensory data. (for example, "a coin has a 50% chance of coming up heads"). In terms of predicting things in the external world, it works very well. Where it breaks down is when you try to apply it to reasoning about your own internal thought processes. It was never intended to do this. As statistician Andrew Gelman correctly points out, it is simply invalid to try to assign probabilities to mathematical statements or theories, for instance. Can an alternative mathematical framework be developed, one more in keeping with the ideas of David Deutsch and the coherence theory of knowledge? I believe the answer is yes, and I am going to sketch the basic ideas for such a framework. The basic idea is to separate out levels of abstraction when reasoning (or equivalently, levels of recursion). In my proposed framework, there are 3 levels, and each level gets its own measure of 'truth-value'. All reasoning must terminate in a Boolean truth value (True/False) at the base level but the idea is that different forms of reasoning correspond to different levels of abstraction. 1st level: Boolean logic (True/False) 2nd level: Probability value (0-1) 3rd level: Conceptual coherence (categorization measure) For full reflection, you need three different numbers: a Boolean value (T/F) at the base, a probability value (0-1) at the next level of abstraction, and an entirely new measure called conceptual coherence at the highest of abstraction. As a rough working definition of conceptual coherence, I would define it thusly; "The degree to which a concept coheres with (integrates with) the overall world-model." It should now be clear what's wrong with Bayesianism! It only gets us to the 2nd level abstraction! There is not just uncertainty about our own knowledge of the world (probability), there is another meta-level of uncertainly; uncertainty about our own reasoning processes, or logical uncertainty. Bayesianism can't help us here. Conceptual coherence can. Lets see how: All statements of the form: ‘outcome x has probability y’ can be converted into statements about conceptual coherence, simply by redefining ‘x’ as a concept in a world-model. Then the correct form of logical expression is: ‘concept x has coherence value y’. The idea is that probability values are just special cases of coherence (the notion of coherence is more general than the notion of probabilities). To conclude, conceptual coherence is the degree with which a concept is integrated with the rest of your world-model, and I think it accurately captures in mathematical terms the ideas that Deutsch was trying express, and is a more powerful method of reasoning than Bayesianism.