8 ms·

Trapped priors as a basic problem of rationality

131 points by yamrzou 6y ago

jhardy54 6y ago
Interesting perspective, thanks for sharing.
tshaddox 6y ago
So, can you stick the cynophobe in the room with the well-behaved Rottweiler and also give them some drug that reduces their anxiety just while they’re in the room?
- adav 6y ago
  Is the Rottweiler particularly anxious to be in the room?
- schoen 6y ago
  There's been recent research on a drug that can help make exposure therapy faster and more effective in just this way, although I can't remember what it's called and wasn't able to find it with a quick search. :-( Edit: I found it with a less-quick search; it's called D-cycloserine. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4081005/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4081005/ I think there is still other research on using other memory-affecting and/or emotion-affecting medications to facilitate exposure therapy.
philipkglass 6y ago
Rationality in service to what end, though? You can't just build some complicated instruments to record measurements of the ideal polity, the way you can refine measurements of the electron's mass. Most political disagreements are about values more than wonkish policymaking. But in fact many political zealots never accept reality. It's not just that they're inherently skeptical of what the other party says. It's that even when something is proven beyond a shadow of a doubt, they still won't believe it. This could describe cognitive error [1] or it could describe someone who's using deontological reasoning more than consequentialist reasoning. (And perhaps lacking the vocabulary to say "consequentialist arguments won't sway my values.") It is probably a good strategy to refuse to be talked into positions against your fundamental values by mere evidence. I'm joking but also serious. I have a vague sense that torturing suspects doesn't work to prevent terrorist attacks, but that's not the root of my opposition to torture. Maybe clever torturers could make it work more often than not. That wouldn't sway me, because torturing suspects is against my values. Documenting how well it can "work" won't make me reconsider. [1] Further food for thought: "cognitive error" and "revealed preference" may also be different names for the same thing. Are they experimentally distinguishable?
- catlifeonmars 6y ago
  I believe that the author makes the basic assumption that rationality is the end.
- renewiltord 6y ago
  Rationality as a tool to serve yourself. A lot of SSC and AC10 is around this idea that rational intelligent thinking with self awareness of bias allows you to model the world accurately. As for the rest, yes, I agree with you. Often, after having considered a great deal of evidence, I condense the finding into some strength of belief about something. Now, absent the original evidence I have a position. This is usually fine since I actually lack the ability to summon all evidence on demand. It's okay, though, since I (and I suspect most people intuitively) have both a likelihood notion and a certainty in likelihood notion analogous to a Confidence Interval and P-value, if you will. So I might hold the belief that GME is going to be between 120 and infinity on Jan 13 2022 and have a certainty in that belief that is like 10% but lose the evidence that made me think that. This is also sort of a failure mode for fast Aumann Agreement even in co-incentivized individuals - the fact that memory is short. So the problem is that my certainty holding is not well done. Thanks for the thought prompt.
  - astrange 6y ago
    I've always found it odd that the people called "rationalists" are not rationalist. Talking about evidence and realistic priors all the time is good (they don't do enough evidence gathering though) but it is not rationalism, it is empiricism. Surely a rationalist would be into logic and actively refuse to read studies, like the Austrian economists did.
    - renewiltord 6y ago
      It's just a terminology thing. Sort of like how atheist actuaries still account for Acts of God.
    - spiderjerusalem 6y ago
      The word 'rational' is one of the most abused in western intellectual history. Hegel considered his system 'rational' and I think it is fairly self-evident that that has nothing to do with these internet 'rationalists'.
BaseS4 6y ago
Every time I see one of these neuromyths about how the brain works being applied to politics, it always bends over backwards to make sure groups of people, when given huge media and financial resources and the motive to influence the decision-making behavior of hundreds of millions of people, are not liable, accountable, or involved in how effective their influence campaigns are. Example: > Segments of Group: Cut off your children's genitals. > Person: Well, uh... okay... for the good of the cause, I guess. > Other Segments of Group: We didn't tell you to cut off your children's genitals. > Apologists of Group: Anyone who would recommend such a thing would NEVER be part of our group! Those groups are absolutely liable for their influence campaigns. It's just that individuals have absolutely no recourse to hold these groups accountable. For proof of that, notice the frequency of active words like "revenge" and "wrath" decrease over time for neutral, passive, and nonthreatening words like "balance" and "equality". People are so demoralized, they just get fed bogus neuromyths about political behavior and accept them because they known there is fuck all they can do about it anyways other than acquiescence and servitude. https://books.google.com/ngrams/graph?content=Revenge%2Cvengeance%2Cwrath%2Cvengeful%2Cbalance%2Cequality&year_start=1800&year_end=2019&corpus=26&smoothing=3&direct_url=t1%3B%2CRevenge%3B%2Cc0%3B.t1%3B%2Cvengeance%3B%2Cc0%3B.t1%3B%2Cwrath%3B%2Cc0%3B.t1%3B%2Cvengeful%3B%2Cc0%3B.t1%3B%2Cbalance%3B%2Cc0%3B.t1%3B%2Cequality%3B%2Cc0 https://books.google.com/ngrams/graph?content=Revenge%2Cveng...
sideshowb 6y ago
Hey this voice sounds familiar... Yay!
tshaddox 6y ago
The problem that I always have with this author’s material and most material in the “rationalist movement” is that after making very valid points about biases, irrational behavior, etc., he seems to leave things at some sort of implicit epistemological nihilism. Since people on the left and the right in American politics both have biases, and both believe their beliefs just as strongly as the other side, apparently we can’t discern anything about their political views other than how their views make them feel? That’s the vibe I’m getting from stuff like this: > I've had arguments with people who believe that no pro-life conservative really cares about fetuses, they just want to punish women for being sluts by denying them control over their bodies. And I've had arguments with people who believe that no pro-lockdown liberal really cares about COVID deaths, they just like the government being able to force people to wear masks as a sign of submission. Once you're at the point where all these things sound plausible, you are doomed. Apparently you’re doomed if you believe one of those things describes reality more than the other? They’re both extreme generalizations, obviously, and are defined imprecisely (“no true Scotsman”), but couldn’t it be that one of those two groups of people have beliefs that correspond much more with reality? Or is the conclusion that, since this is politics, and political views is intensely personal and prone to bias, that both sides are equally correct? Isn’t it possible that there actually are some powerful political groups who have honest and good intentions and others which do not? Isn’t that possible even if most people have a tendency to believe that their own groups are the good ones, and that belief is just as strong for each group? Is it impossible for me to believe my group’s intentions are honest and good, and you believe just as strongly that your group’s intentions are honest and good, but that only one of us is correct? Is knowledge nothing more than a feeling of certainty? Or is knowledge simply impossible?
- jbay808 6y ago
  I think the author only means you're "doomed" in the sense that "your fate is sealed": you'll find it very hard to change your mind in the face of contrary evidence, because your observations will start to feel like they always reinforce your conclusion.
  - tshaddox 6y ago
catlifeonmars 6y ago
Is it just me or does anyone else find the “just-so” style of explaining phenomenon in this article irritating? Edit: Less snarky response here. The “just-so” style that the author uses to explain psychological phenomena appeals to the readers’ cognitive biases and avoids confronting the reader with the more rigorous, mechanistic (perhaps evidence-based?) explanations that the same author proposes as a way to combat said cognitive biases. This strikes me as mildly ironic.
- BaseS4 6y ago
  It's a neuromyth being applied to political domains so that the believers of political narratives don't have to feel accountable for anything.
- heavyset_go 6y ago
  > The “just-so” style that the author uses to explain psychological phenomena appeals to the readers’ cognitive biases and avoids confronting the reader with the more rigorous, mechanistic (perhaps evidence-based?) explanations that the same author proposes as a way to combat said cognitive biases. This strikes me as mildly ironic. This is endemic to the rationalist community. It's actually quite humorous once you begin to notice it.
- AliceWong 6y ago
  What do you mean by "just-so" style? I'm not familiar with the term.
  - catlifeonmars 6y ago
    > In science and philosophy, a just-so story is an untestable narrative explanation for a cultural practice, a biological trait, or behavior of humans or other animals. The pejorative[1] nature of the expression is an implicit criticism that reminds the listener of the essentially fictional and unprovable nature of such an explanation. Such tales are common in folklore and mythology (where they are known as etiological myths—see etiology). https://en.wikipedia.org/wiki/Just-so_story https://en.wikipedia.org/wiki/Just-so_story
jbay808 6y ago
A lot of the time, this might not be a property of the update process so much as the prior itself. A conspiracy theory is often something that can explain any lack of observational evidence for itself (because it's covered up!). If the prior on the conspiracy being true is too high, the update on contrary evidence will be very weak just because the conspiracy theory makes weak predictions about upcoming observations, so it loses probability mass very slowly as a whole, with probability mass mostly just being shifted within the context of the theory. ("They're covering up aliens!" --> no evidence of this after fifty years? --> "It's worse than I thought; they must be using mind control!") The only cure for this might be preventative -- just maintaining a strong doubt on deep conspiracy theories. But that also means resigning yourself to never believing in one before it's revealed. If it turns out you live in a reality where the government is using mind control to conceal aliens, you'll not be the one who realizes it before everyone else.
- yorwba 6y ago
  > the conspiracy theory makes weak predictions about upcoming observations, so it loses probability mass very slowly as a whole That may be the case for conspiracy theories, but I'd expect the theory that dogs are dangerous to make much stronger predictions. On the other hand, maybe people who're afraid of dogs don't expect them to behave differently, they just think that everything dogs do is scary (which is true, since they're scared by everything dogs do.)
  - jbay808 6y ago
    I think you're right; it could be either. In this example, if their fear of dogs causes them to misperceive a dog's excitement as anger, then it's closer to the scenario in the article. But if their paranoia is such that they perceive the dog is acting friendly but interpret that as the dog luring them into a sense of safety to bite them later, then it's more like the conspiracy theory case.
  - Siira 6y ago
    It’s about power; Dogs being violent isn’t super rare, and if a dog decides to get violent, I have no control over it. The probability of the danger is low, but it is sufficiently bad even in mild cases that I do not want to take any chances.
analog31 6y ago
Has Bayesian verbiage become a form of virtue signaling? Don't get me wrong. I took statistics in college, and we covered Bayesian methods. They can't be controversial because Bayes' Theorem is a theorem, meaning it's proven. But the attempt to apply its terminology to lengthy but "soft" arguments seems like pure clutter.
- andrewflnr 6y ago
  I think it's been common in the rationalist community for a while, in the 5-10 years that I've been aware of them.
- fractionalhare 6y ago
  I don't know if virtue signaling is the right term. The better term might be "shibboleth" for so-called rationalists. It's as if you took the kids who loved lists of logical fallacies and told them they could resolve all disputes if they just reduced them to first principles and used words like "prior." Nevermind that the first principles lose all context of the original issue - we're in a world of pure logic, where we can use theorems! To the rationalist community, everything looks like a good opportunity to write a lengthy essay reinventing the wheel with Bayes theorem. It's a cargo cult of basic statistics dressing up otherwise uninteresting arguments. It's very tedious for people like myself who actually studied statistics and apply it professionally. Bayes theorem is barely even a theorem - it's literally a basic rearrangement of the axioms of probability. It's not some supreme revelation that imbues arguments with more credibility. Most of the time I see it invoked in discussion, it's not even rigorously quantified. You can choose arbitrary priors, so at the end of the day you never arrive at an objective truth. It's profundity for its own sake, and nothing is really achieved.
- catlifeonmars 6y ago
  I can’t speak to how others interpret it, but to me, I read it as a way to apply a pretense of mathematical rigor in otherwise unrigorous contexts (like psychology or behavioral economics).
- sega_sai 6y ago
  No, I don't think it is. Separately from rationalist community, there is substantial amount of work showing that Bayesian updating and reasoning is in some sense the optimal way of operating under uncertainty. It is also certainly true that garbage in and garbage out, i.e. if you feed garbage inputs, you'll get garbage out (Bayesian or not), but the advantage in my opinion of having a framework for making decisions in cases of uncertainties is extremely useful. It is often hard to really apply this for ordinary life situations, (and maybe often not worth it), but even then it's useful to think about priors/evidence, even if you don't particularly care about the final number.
getpost 6y ago
> I wonder if it's the equivalent of making trauma victims describe the traumatic event in detail; an attempt to give higher weight to the raw experience pathway compared to the prior pathway. > The other promising source of hope is psychedelics. For me, what this part if the essay points to is that everyone needs to do their own work, in whatever context, and there's no getting around that. Experts, authorities, and managers need to find the line between facilitating a process and doing the process themselves. You can't make anyone believe anything or do anything. You can only hope to facilitate the conditions where helpful beliefs and accomplishments arise. I hope this is one significant cultural shift during my lifetime. The 20th century was about creating expertise and giving that expertise authority, which is just another expression of the dominator paradigm.
Gatsky 6y ago
I wish the scroll bar range conformed to the length of the article rather than the length of the article plus comments. SSC/ASC articles are already quite long. I was almost put off reading by the languid progression of the scroll bar marker.
jonnycomputer 6y ago
The argument here is insufficiently clear on the mechanism by which priors get trapped. I'm thinking the argument goes something like this: a Bayesian updater takes in new evidence, and the adjusts their priors based on that evidence. Crucially, the information that is fed back into the Bayesian machine should be independent of the prior. People do not have direct access to evidence. They only have their perception of that evidence, and the content of those perceptions depend in various complicated ways on people's priors. Therefore, evidence that should move people's priors may move them less than they should. Sounds plausible. I think reinforcement learning algorithms like temporal difference learning are probably a more accurate description of how people learn tha Bayesian update, but the issue described would apply regardless.
ilaksh 6y ago
Its weird to me that people are not familiar with the concept of worldview or beliefs and that they have an often overwhelming influence over thinking. And I think that everyone has a worldview. No one is impervious to this or truly non-partisan. Some people may be less partisan in some ways. But they still have worldviews that influence their thinking, possibly in somewhat different areas.

131 points by yamrzou 6y ago

Interesting perspective, thanks for sharing.

So, can you stick the cynophobe in the room with the well-behaved Rottweiler and also give them some drug that reduces their anxiety just while they’re in the room?

adav 6y ago
Is the Rottweiler particularly anxious to be in the room?
schoen 6y ago
There's been recent research on a drug that can help make exposure therapy faster and more effective in just this way, although I can't remember what it's called and wasn't able to find it with a quick search. :-( Edit: I found it with a less-quick search; it's called D-cycloserine. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4081005/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4081005/ I think there is still other research on using other memory-affecting and/or emotion-affecting medications to facilitate exposure therapy.

philipkglass 6y ago

Rationality in service to what end, though? You can't just build some complicated instruments to record measurements of the ideal polity, the way you can refine measurements of the electron's mass. Most political disagreements are about values more than wonkish policymaking. But in fact many political zealots never accept reality. It's not just that they're inherently skeptical of what the other party says. It's that even when something is proven beyond a shadow of a doubt, they still won't believe it. This could describe cognitive error [1] or it could describe someone who's using deontological reasoning more than consequentialist reasoning. (And perhaps lacking the vocabulary to say "consequentialist arguments won't sway my values.") It is probably a good strategy to refuse to be talked into positions against your fundamental values by mere evidence. I'm joking but also serious. I have a vague sense that torturing suspects doesn't work to prevent terrorist attacks, but that's not the root of my opposition to torture. Maybe clever torturers could make it work more often than not. That wouldn't sway me, because torturing suspects is against my values. Documenting how well it can "work" won't make me reconsider. [1] Further food for thought: "cognitive error" and "revealed preference" may also be different names for the same thing. Are they experimentally distinguishable?

catlifeonmars 6y ago
I believe that the author makes the basic assumption that rationality is the end.
renewiltord 6y ago
Rationality as a tool to serve yourself. A lot of SSC and AC10 is around this idea that rational intelligent thinking with self awareness of bias allows you to model the world accurately. As for the rest, yes, I agree with you. Often, after having considered a great deal of evidence, I condense the finding into some strength of belief about something. Now, absent the original evidence I have a position. This is usually fine since I actually lack the ability to summon all evidence on demand. It's okay, though, since I (and I suspect most people intuitively) have both a likelihood notion and a certainty in likelihood notion analogous to a Confidence Interval and P-value, if you will. So I might hold the belief that GME is going to be between 120 and infinity on Jan 13 2022 and have a certainty in that belief that is like 10% but lose the evidence that made me think that. This is also sort of a failure mode for fast Aumann Agreement even in co-incentivized individuals - the fact that memory is short. So the problem is that my certainty holding is not well done. Thanks for the thought prompt.
- astrange 6y ago
  I've always found it odd that the people called "rationalists" are not rationalist. Talking about evidence and realistic priors all the time is good (they don't do enough evidence gathering though) but it is not rationalism, it is empiricism. Surely a rationalist would be into logic and actively refuse to read studies, like the Austrian economists did.
  - renewiltord 6y ago
    It's just a terminology thing. Sort of like how atheist actuaries still account for Acts of God.
  - spiderjerusalem 6y ago
    The word 'rational' is one of the most abused in western intellectual history. Hegel considered his system 'rational' and I think it is fairly self-evident that that has nothing to do with these internet 'rationalists'.

BaseS4 6y ago

Every time I see one of these neuromyths about how the brain works being applied to politics, it always bends over backwards to make sure groups of people, when given huge media and financial resources and the motive to influence the decision-making behavior of hundreds of millions of people, are not liable, accountable, or involved in how effective their influence campaigns are. Example: > Segments of Group: Cut off your children's genitals. > Person: Well, uh... okay... for the good of the cause, I guess. > Other Segments of Group: We didn't tell you to cut off your children's genitals. > Apologists of Group: Anyone who would recommend such a thing would NEVER be part of our group! Those groups are absolutely liable for their influence campaigns. It's just that individuals have absolutely no recourse to hold these groups accountable. For proof of that, notice the frequency of active words like "revenge" and "wrath" decrease over time for neutral, passive, and nonthreatening words like "balance" and "equality". People are so demoralized, they just get fed bogus neuromyths about political behavior and accept them because they known there is fuck all they can do about it anyways other than acquiescence and servitude. https://books.google.com/ngrams/graph?content=Revenge%2Cvengeance%2Cwrath%2Cvengeful%2Cbalance%2Cequality&year_start=1800&year_end=2019&corpus=26&smoothing=3&direct_url=t1%3B%2CRevenge%3B%2Cc0%3B.t1%3B%2Cvengeance%3B%2Cc0%3B.t1%3B%2Cwrath%3B%2Cc0%3B.t1%3B%2Cvengeful%3B%2Cc0%3B.t1%3B%2Cbalance%3B%2Cc0%3B.t1%3B%2Cequality%3B%2Cc0 https://books.google.com/ngrams/graph?content=Revenge%2Cveng...

sideshowb 6y ago

Hey this voice sounds familiar... Yay!

tshaddox 6y ago

The problem that I always have with this author’s material and most material in the “rationalist movement” is that after making very valid points about biases, irrational behavior, etc., he seems to leave things at some sort of implicit epistemological nihilism. Since people on the left and the right in American politics both have biases, and both believe their beliefs just as strongly as the other side, apparently we can’t discern anything about their political views other than how their views make them feel? That’s the vibe I’m getting from stuff like this: > I've had arguments with people who believe that no pro-life conservative really cares about fetuses, they just want to punish women for being sluts by denying them control over their bodies. And I've had arguments with people who believe that no pro-lockdown liberal really cares about COVID deaths, they just like the government being able to force people to wear masks as a sign of submission. Once you're at the point where all these things sound plausible, you are doomed. Apparently you’re doomed if you believe one of those things describes reality more than the other? They’re both extreme generalizations, obviously, and are defined imprecisely (“no true Scotsman”), but couldn’t it be that one of those two groups of people have beliefs that correspond much more with reality? Or is the conclusion that, since this is politics, and political views is intensely personal and prone to bias, that both sides are equally correct? Isn’t it possible that there actually are some powerful political groups who have honest and good intentions and others which do not? Isn’t that possible even if most people have a tendency to believe that their own groups are the good ones, and that belief is just as strong for each group? Is it impossible for me to believe my group’s intentions are honest and good, and you believe just as strongly that your group’s intentions are honest and good, but that only one of us is correct? Is knowledge nothing more than a feeling of certainty? Or is knowledge simply impossible?

jbay808 6y ago
I think the author only means you're "doomed" in the sense that "your fate is sealed": you'll find it very hard to change your mind in the face of contrary evidence, because your observations will start to feel like they always reinforce your conclusion.
- tshaddox 6y ago

catlifeonmars 6y ago

Is it just me or does anyone else find the “just-so” style of explaining phenomenon in this article irritating? Edit: Less snarky response here. The “just-so” style that the author uses to explain psychological phenomena appeals to the readers’ cognitive biases and avoids confronting the reader with the more rigorous, mechanistic (perhaps evidence-based?) explanations that the same author proposes as a way to combat said cognitive biases. This strikes me as mildly ironic.

BaseS4 6y ago
It's a neuromyth being applied to political domains so that the believers of political narratives don't have to feel accountable for anything.
heavyset_go 6y ago
> The “just-so” style that the author uses to explain psychological phenomena appeals to the readers’ cognitive biases and avoids confronting the reader with the more rigorous, mechanistic (perhaps evidence-based?) explanations that the same author proposes as a way to combat said cognitive biases. This strikes me as mildly ironic. This is endemic to the rationalist community. It's actually quite humorous once you begin to notice it.
AliceWong 6y ago
What do you mean by "just-so" style? I'm not familiar with the term.
- catlifeonmars 6y ago
  > In science and philosophy, a just-so story is an untestable narrative explanation for a cultural practice, a biological trait, or behavior of humans or other animals. The pejorative[1] nature of the expression is an implicit criticism that reminds the listener of the essentially fictional and unprovable nature of such an explanation. Such tales are common in folklore and mythology (where they are known as etiological myths—see etiology). https://en.wikipedia.org/wiki/Just-so_story https://en.wikipedia.org/wiki/Just-so_story

jbay808 6y ago

A lot of the time, this might not be a property of the update process so much as the prior itself. A conspiracy theory is often something that can explain any lack of observational evidence for itself (because it's covered up!). If the prior on the conspiracy being true is too high, the update on contrary evidence will be very weak just because the conspiracy theory makes weak predictions about upcoming observations, so it loses probability mass very slowly as a whole, with probability mass mostly just being shifted within the context of the theory. ("They're covering up aliens!" --> no evidence of this after fifty years? --> "It's worse than I thought; they must be using mind control!") The only cure for this might be preventative -- just maintaining a strong doubt on deep conspiracy theories. But that also means resigning yourself to never believing in one before it's revealed. If it turns out you live in a reality where the government is using mind control to conceal aliens, you'll not be the one who realizes it before everyone else.

yorwba 6y ago
> the conspiracy theory makes weak predictions about upcoming observations, so it loses probability mass very slowly as a whole That may be the case for conspiracy theories, but I'd expect the theory that dogs are dangerous to make much stronger predictions. On the other hand, maybe people who're afraid of dogs don't expect them to behave differently, they just think that everything dogs do is scary (which is true, since they're scared by everything dogs do.)
- jbay808 6y ago
  I think you're right; it could be either. In this example, if their fear of dogs causes them to misperceive a dog's excitement as anger, then it's closer to the scenario in the article. But if their paranoia is such that they perceive the dog is acting friendly but interpret that as the dog luring them into a sense of safety to bite them later, then it's more like the conspiracy theory case.
- Siira 6y ago
  It’s about power; Dogs being violent isn’t super rare, and if a dog decides to get violent, I have no control over it. The probability of the danger is low, but it is sufficiently bad even in mild cases that I do not want to take any chances.

analog31 6y ago

Has Bayesian verbiage become a form of virtue signaling? Don't get me wrong. I took statistics in college, and we covered Bayesian methods. They can't be controversial because Bayes' Theorem is a theorem, meaning it's proven. But the attempt to apply its terminology to lengthy but "soft" arguments seems like pure clutter.

andrewflnr 6y ago
I think it's been common in the rationalist community for a while, in the 5-10 years that I've been aware of them.
fractionalhare 6y ago
I don't know if virtue signaling is the right term. The better term might be "shibboleth" for so-called rationalists. It's as if you took the kids who loved lists of logical fallacies and told them they could resolve all disputes if they just reduced them to first principles and used words like "prior." Nevermind that the first principles lose all context of the original issue - we're in a world of pure logic, where we can use theorems! To the rationalist community, everything looks like a good opportunity to write a lengthy essay reinventing the wheel with Bayes theorem. It's a cargo cult of basic statistics dressing up otherwise uninteresting arguments. It's very tedious for people like myself who actually studied statistics and apply it professionally. Bayes theorem is barely even a theorem - it's literally a basic rearrangement of the axioms of probability. It's not some supreme revelation that imbues arguments with more credibility. Most of the time I see it invoked in discussion, it's not even rigorously quantified. You can choose arbitrary priors, so at the end of the day you never arrive at an objective truth. It's profundity for its own sake, and nothing is really achieved.
catlifeonmars 6y ago
I can’t speak to how others interpret it, but to me, I read it as a way to apply a pretense of mathematical rigor in otherwise unrigorous contexts (like psychology or behavioral economics).
sega_sai 6y ago
No, I don't think it is. Separately from rationalist community, there is substantial amount of work showing that Bayesian updating and reasoning is in some sense the optimal way of operating under uncertainty. It is also certainly true that garbage in and garbage out, i.e. if you feed garbage inputs, you'll get garbage out (Bayesian or not), but the advantage in my opinion of having a framework for making decisions in cases of uncertainties is extremely useful. It is often hard to really apply this for ordinary life situations, (and maybe often not worth it), but even then it's useful to think about priors/evidence, even if you don't particularly care about the final number.

getpost 6y ago

> I wonder if it's the equivalent of making trauma victims describe the traumatic event in detail; an attempt to give higher weight to the raw experience pathway compared to the prior pathway. > The other promising source of hope is psychedelics. For me, what this part if the essay points to is that everyone needs to do their own work, in whatever context, and there's no getting around that. Experts, authorities, and managers need to find the line between facilitating a process and doing the process themselves. You can't make anyone believe anything or do anything. You can only hope to facilitate the conditions where helpful beliefs and accomplishments arise. I hope this is one significant cultural shift during my lifetime. The 20th century was about creating expertise and giving that expertise authority, which is just another expression of the dominator paradigm.

Gatsky 6y ago

I wish the scroll bar range conformed to the length of the article rather than the length of the article plus comments. SSC/ASC articles are already quite long. I was almost put off reading by the languid progression of the scroll bar marker.

jonnycomputer 6y ago

The argument here is insufficiently clear on the mechanism by which priors get trapped. I'm thinking the argument goes something like this: a Bayesian updater takes in new evidence, and the adjusts their priors based on that evidence. Crucially, the information that is fed back into the Bayesian machine should be independent of the prior. People do not have direct access to evidence. They only have their perception of that evidence, and the content of those perceptions depend in various complicated ways on people's priors. Therefore, evidence that should move people's priors may move them less than they should. Sounds plausible. I think reinforcement learning algorithms like temporal difference learning are probably a more accurate description of how people learn tha Bayesian update, but the issue described would apply regardless.

ilaksh 6y ago

Its weird to me that people are not familiar with the concept of worldview or beliefs and that they have an often overwhelming influence over thinking. And I think that everyone has a worldview. No one is impervious to this or truly non-partisan. Some people may be less partisan in some ways. But they still have worldviews that influence their thinking, possibly in somewhat different areas.