5 ms·

At 40 years old, what’s clear to me is that while people have some curiosities they pursue, most have no desire to be rigorous about epistemological questions s

At 40 years old, what’s clear to me is that while people have some curiosities they pursue, most have no desire to be rigorous about epistemological questions surrounding their curiosities. This extends well into people’s professional disciplines in my experience. So even for the small percentage of people who have some narrow focus of inquiry (usually at work) they only have the desire or ability to evaluate the first or second order inputs and effects. In general this isn’t a problem for day to day life. The packaged chicken on the shelf fits perfectly into recipe, and there is little to no thought of any other externalities (what systems am I “voting with my dollars” to support? What is the relative difference in supply chain between two options) However if you are going to try and align your conceptions of how systems/environments work and what you desire for future state, then you do have to do the system decomposition and that’s actually non trivial. So there’s no real “market” for epistemological rigor outside of a small group of people who come across as unreasonable in their demand for rigor, so this is the likely result of a system that is built for personal or organizational success and not an ego-free search for “truth” via intentional rigor. Luckily the small group continues to grow as a total number

pessimizer 3y ago
It takes too much work to verify everything in every field personally, you wouldn't be able to move at all. The problem is a complete lack of interest in any tools that would inform people about this, because entire industries and fields are built by bad papers. We seem to have a lot of hypertext systems that keep track of citations. Seems like it would be cheap at this point to put a black mark on papers that don't replicate, and to propagate that black mark to every paper that references the bad paper without also referencing the failed replication. Put a big red X over the abstracts.
- smallmancontrov 3y ago
  > It takes too much work to verify everything in every field personally, you wouldn't be able to move at all. Does it? The rate of paper production is so high that it has become a logistical problem in and of itself and wild goose chases are common. Surely some of that energy could be profitably redirected into replication and consolidation. The black mark system could work but seems like a recipe for some nasty politics. I tend to think that baking replication into the culture might be a better approach -- science advanced beyond secret-hoarding when information sharing became institutionally valued, and I believe that it could advance beyond p-hacking if replication became institutionally valued. Fund & cite replication efforts and the rest will follow.
- btrettel 3y ago
  > It takes too much work to verify everything in every field personally, you wouldn't be able to move at all. Yes and no. The scale of the checking problem is formidable, as you're aware. However, many claims in scientific papers are weak. Even highly cited papers can contain nonsense that doesn't stand up to mild scrutiny. Given that, I think progress can be made towards the problem if every researcher simply spent more time checking previous work. Checking everything is impossible, but progress can be made by prioritizing, dividing the work up among everyone, and doing checks that don't take much time. As an example of a highly cited paper with easily noticed major issues, consider this comment I posted on PubPeer: https://pubpeer.com/publications/95455FA4147A9CBD5EAA5185D21DAC https://pubpeer.com/publications/95455FA4147A9CBD5EAA5185D21... The first version of this paper was published in 1991 and has over 300 citations. It's still cited to this day. Despite this, in 2016 I read the later journal version of the paper and pretty quickly realized that the first part of the paper was basically nonsense. You don't need to look hard to realize that either! The problem can be found by simple spot checks, looking at the trends of the equation. (I can't quite recall, but I think that's how I figured out that the model was wrong in the first place.) My paper refuting the model to date has zero citations that aren't from me. The closest I've received so far was an email from someone who was curious that I was apparently the first to notice the problem, over 20 years after the first publication of the model.
hiAndrewQuinn 3y ago
A related point is made in Michael Huemer's _In Praise of Passivity_, where it is used to argue that almost everyone would be better off simply not getting involved in political processes like voting at all. [1] I think it generalizes to most domains. Rigorous knowledge in any domain is both difficult to obtain, and it usually commands a small premium compared to the other things you could have sunk your time and attention into. I see at least two options for getting around this: - Wirehead yourself into thinking truth is beauty and beauty truth, and then tilt at what many others will consider windmills (the fools!). At its best this is probably what folks like Isaac Newton were up to. - Go all in on the profit motive, embrace competition as a refining fire to force you to be more rigorous or go broke trying. This reduces the problem to knowledge being "just" really hard to obtain. There may well be more, those are just the two most obvious ones to my eyes. [1]: https://bazhum.muzhp.pl/media/files/Studia_Humana/Studia_Humana-r2012-t-n2/Studia_Humana-r2012-t-n2-s12-28/Studia_Humana-r2012-t-n2-s12-28.pdf https://bazhum.muzhp.pl/media/files/Studia_Humana/Studia_Hum...
- AndrewKemendo 3y ago
  I can’t think of anything worse than those approaches It’s pessimistic at the extreme and throws one’s hands up to nihilism I prefer pushing through that nihilism and into realizing that absurdity of the universe is not a fixed barrier but a challenge to solve for the entirety of the universe
  - hiAndrewQuinn 3y ago
    I guess that's a third approach you can take, if you're already a nihilist on some level. I'm not, so I find the message of finding niches people haven't pushed hard enough into to make the world a better place (and maybe make some money while you're at it) to be pretty uplifting.
faeriechangling 3y ago
In my experience, there's little to no reward for rigor and people mostly invoke it when somebody says something they don't want to hear. People are impressed enough if you cite some bullcrap even if it's been refuted since they themselves don't have the rigor to actually question if what you cited is correct. They usually don't have the rigor to even check your original source in the first place.
- AndrewKemendo 3y ago
  It's an existential problem for the species that we can encode into our structures conceptions of the way the universe works that are measurably wrong Bloodletting is well understood that it doesn't work but there was a time where the most trusted people performed that Why is that even possible? There was absolutely no epistemic proof that bloodletting was associated with a falsifiable hypothesis which would lead to improved health outcomes - in fact it was all based on these "humors" in the body that needed to be "balanced" which makes no mathematical sense whatsoever. Even then, as early as the 1400s there were scientists saying "This is actually bad" - and yet George Washington had bloodletting done on him in his dying repose at the end of the 1700s 300 years of an activity that is bad for you that was encoded into medical practice The conclusion we should come to is that MOST of our assumptions about the structure of the world and how to optimize within it, are based on literally nothing but aggregated social velocity, instead of claims that are measurable and testable
  - faeriechangling 3y ago
    The other problem with right I forgot to mention is that Rigor generally means: 1: Refusing to make a firm affirmative statement on an issue, when that's what people want 2: Correcting what you have claimed in the past, which makes you look gullible and indecisive With the example of bloodletting, people wanted to believe a doctor knew how to cure them, and once somebody had done bloodletting even if they had doubts later, they would look like fools if they convinced people they had been harming their patients all along. It's easier and often more rewarding to be a fraud.
    - 3y ago

pessimizer 3y ago

It takes too much work to verify everything in every field personally, you wouldn't be able to move at all. The problem is a complete lack of interest in any tools that would inform people about this, because entire industries and fields are built by bad papers. We seem to have a lot of hypertext systems that keep track of citations. Seems like it would be cheap at this point to put a black mark on papers that don't replicate, and to propagate that black mark to every paper that references the bad paper without also referencing the failed replication. Put a big red X over the abstracts.

smallmancontrov 3y ago
> It takes too much work to verify everything in every field personally, you wouldn't be able to move at all. Does it? The rate of paper production is so high that it has become a logistical problem in and of itself and wild goose chases are common. Surely some of that energy could be profitably redirected into replication and consolidation. The black mark system could work but seems like a recipe for some nasty politics. I tend to think that baking replication into the culture might be a better approach -- science advanced beyond secret-hoarding when information sharing became institutionally valued, and I believe that it could advance beyond p-hacking if replication became institutionally valued. Fund & cite replication efforts and the rest will follow.
btrettel 3y ago
> It takes too much work to verify everything in every field personally, you wouldn't be able to move at all. Yes and no. The scale of the checking problem is formidable, as you're aware. However, many claims in scientific papers are weak. Even highly cited papers can contain nonsense that doesn't stand up to mild scrutiny. Given that, I think progress can be made towards the problem if every researcher simply spent more time checking previous work. Checking everything is impossible, but progress can be made by prioritizing, dividing the work up among everyone, and doing checks that don't take much time. As an example of a highly cited paper with easily noticed major issues, consider this comment I posted on PubPeer: https://pubpeer.com/publications/95455FA4147A9CBD5EAA5185D21DAC https://pubpeer.com/publications/95455FA4147A9CBD5EAA5185D21... The first version of this paper was published in 1991 and has over 300 citations. It's still cited to this day. Despite this, in 2016 I read the later journal version of the paper and pretty quickly realized that the first part of the paper was basically nonsense. You don't need to look hard to realize that either! The problem can be found by simple spot checks, looking at the trends of the equation. (I can't quite recall, but I think that's how I figured out that the model was wrong in the first place.) My paper refuting the model to date has zero citations that aren't from me. The closest I've received so far was an email from someone who was curious that I was apparently the first to notice the problem, over 20 years after the first publication of the model.

hiAndrewQuinn 3y ago

A related point is made in Michael Huemer's _In Praise of Passivity_, where it is used to argue that almost everyone would be better off simply not getting involved in political processes like voting at all. [1] I think it generalizes to most domains. Rigorous knowledge in any domain is both difficult to obtain, and it usually commands a small premium compared to the other things you could have sunk your time and attention into. I see at least two options for getting around this: - Wirehead yourself into thinking truth is beauty and beauty truth, and then tilt at what many others will consider windmills (the fools!). At its best this is probably what folks like Isaac Newton were up to. - Go all in on the profit motive, embrace competition as a refining fire to force you to be more rigorous or go broke trying. This reduces the problem to knowledge being "just" really hard to obtain. There may well be more, those are just the two most obvious ones to my eyes. [1]: https://bazhum.muzhp.pl/media/files/Studia_Humana/Studia_Humana-r2012-t-n2/Studia_Humana-r2012-t-n2-s12-28/Studia_Humana-r2012-t-n2-s12-28.pdf https://bazhum.muzhp.pl/media/files/Studia_Humana/Studia_Hum...

AndrewKemendo 3y ago
I can’t think of anything worse than those approaches It’s pessimistic at the extreme and throws one’s hands up to nihilism I prefer pushing through that nihilism and into realizing that absurdity of the universe is not a fixed barrier but a challenge to solve for the entirety of the universe
- hiAndrewQuinn 3y ago
  I guess that's a third approach you can take, if you're already a nihilist on some level. I'm not, so I find the message of finding niches people haven't pushed hard enough into to make the world a better place (and maybe make some money while you're at it) to be pretty uplifting.

faeriechangling 3y ago

In my experience, there's little to no reward for rigor and people mostly invoke it when somebody says something they don't want to hear. People are impressed enough if you cite some bullcrap even if it's been refuted since they themselves don't have the rigor to actually question if what you cited is correct. They usually don't have the rigor to even check your original source in the first place.

AndrewKemendo 3y ago
It's an existential problem for the species that we can encode into our structures conceptions of the way the universe works that are measurably wrong Bloodletting is well understood that it doesn't work but there was a time where the most trusted people performed that Why is that even possible? There was absolutely no epistemic proof that bloodletting was associated with a falsifiable hypothesis which would lead to improved health outcomes - in fact it was all based on these "humors" in the body that needed to be "balanced" which makes no mathematical sense whatsoever. Even then, as early as the 1400s there were scientists saying "This is actually bad" - and yet George Washington had bloodletting done on him in his dying repose at the end of the 1700s 300 years of an activity that is bad for you that was encoded into medical practice The conclusion we should come to is that MOST of our assumptions about the structure of the world and how to optimize within it, are based on literally nothing but aggregated social velocity, instead of claims that are measurable and testable
- faeriechangling 3y ago
  The other problem with right I forgot to mention is that Rigor generally means: 1: Refusing to make a firm affirmative statement on an issue, when that's what people want 2: Correcting what you have claimed in the past, which makes you look gullible and indecisive With the example of bloodletting, people wanted to believe a doctor knew how to cure them, and once somebody had done bloodletting even if they had doubts later, they would look like fools if they convinced people they had been harming their patients all along. It's easier and often more rewarding to be a fraud.
  - 3y ago