7 ms·

I find it sad that over 40 years after this famous speech, which should have been a wakeup call, psychology was ground zero for https://en.wikipedia.org/wiki/Re

by btilly 6y ago

I find it sad that over 40 years after this famous speech, which should have been a wakeup call, psychology was ground zero for https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis. Psychology is making small improvements. But most of what Feynman said was wrong with psychology back then is still wrong today.

sadmann1 6y ago
If you think it's just psychology and not 99% of all publications in all domains including computer science you're quite optimistic
- ta8645 6y ago
  The lack of this being seriously addressed and remedied hurts the reputation of science in general. It also undermines the argument in areas of public policy around say Covid, where "listen to the science" has to overcome the perception that a lot of what passes for science... isn't.
  - ramraj07 6y ago
    In Covid it was far more visibly clear this was and continues to happen. Everyone knows in their hearts that temp check is bullshit but that’s something everyone does even today. The about face on mask policy initially probably still plays a role in people’s perception and inability to follow rules today.
    - jjbinx007 6y ago
      > Everyone knows in their hearts that temp check is bullshit but that’s something everyone does even today. All our local visitor attractions (and I suspect it's the same UK wide) do temp checks on condition of entry. Pointing out the flaws with it is futile as it's the covid equivalent of security theatre. It ticks a box and makes it look like they are taking precautions. It reminds me of those snake oil bomb detectors that are still used in some parts of the world despite the fact that the 'inventors' have been jailed for selling the fraudulent devices: https://www.bbc.co.uk/news/uk-29459896 https://www.bbc.co.uk/news/uk-29459896
    - ACow_Adonis 6y ago
      what's bullshit about temp checks? if it's being sold as a silver bullet that stops all covid (or something else) , well sure, obviously that's bullshit. or is it the idea that it's OK to open everything if you just do temp checks, cause yeah, I can see how that would be bullshit too. But set a high enough threshold (to escape measurement error) and with accurate enough tools, it's one of the few public health things that seems somewhat evidence based: if you've got a verifiable fever, you probably shouldn't be out and mixing with others in confined spaces. hell, it probably made community health sense before COVID. obviously this doesn't stop non-fever, non symptomatic transmission, or people who dope themselves or their kids up to avoid detection, and it doesn't magically get rid of covid, but it picks out the low hanging fruit, and assuming properly calibrated equipment, seems like an obvious public health win to me (as I said, if you've got a fever, you should bloody well be going home/ denied entry to a workplace even in non-covid times).
      - ramraj07 6y ago
        By the time the 75% of patients get fever from Covid they are often fully aware that something’s not right and are probably not leaving home or have already gotten tested, only the crazy ones would still venture out in that stage. Importantly, the reason Covid spread this fast is because the disease seems to attain peak infectiousness before symptom onset and definitely before fever onset. This is why SARS was nipped in the bud with IR scanners while Covid wasn’t. You say that temperature checks don’t hurt but they most definitely do, most places where they do a check, especially in places like India, they think they’ve done enough and don’t enforce strict mask policies or distancing measures any further. If people are forced to stop using temp screening then these places will have to come up with alternative methods of comforting their audience they are taking measures so they would be forced to do actually effective screening methods like taking o2 measurements. So yeah it’s stupid to allow temp checks to pass for any theatre.
        Angostura 6y ago
        >only the crazy ones would still venture out in that stage. Depending on location, that could still be a useful segment to exclude. People takkng their kids on a long-planned and long-promised visit to an attraction, who wake up feeling 'a bit under the weather' may be tempted to go anyway.
  - paganel 6y ago
    Maybe we should try to detangle public policy from science for a while, it hurts both public policy and science.
    - KSteffensen 6y ago
      But shouldn't public policy be informed by (high-quality) science?
      - paganel 6y ago
        Why? Science (especially social sciences) is many, many times wrong, by definition (that's what science does, it gets less and less wrong until it reaches a certain epsilon of "less wrong" which it considers as the "truth"). This is not the fault of science because, again, errors/being wrong are included in its definition, so to speak, the problem is that those errors/wrongs, when applied to the life of real people, with real human lives, will most probably kill them or affect their lives negatively in very big ways, especially so if it's a concerted effort coming from the top. There are countless examples for that, the latest that comes to my mind is the pro-austerity policy imposed by the IMF on Greece, because that's what the science of economics was advising at the time, i.e. austerity. It turned out that that was a mistake (and IMF were quick to acknowledge it), but by that moment the damage had already been done. There's also the example of failed policies imposed by the likes of Robert McNamara and his men (they were mostly men) back in the 1960s, but I didn't catch those times in person, so I only know about those failures via links found on this website (and which I'm too lazy to search for right now).
        bryanrasmussen 6y ago
        there's lots of public policy that is antagonistic to more grounded science than the social sciences, science that while it can be proven wrong at some point is unlikely to be and if it is proven wrong will probably be in small particulars rather than in the large. Evolutionary theory would be one obvious example.
        _0ffh 6y ago
  - youerbt 6y ago
    Well in case of public policy it's rather lack of trust in public policy. And in cases like Covid public policy should be ahead of science anyway, because there is no time to wait.
- renewiltord 6y ago
  The difference with CS is that there's no one to trust. Everything you need is right there. Other disciplines have people fake results. You can't fake a result in CS since people only really attack decidable problems, and you need no empirical results for any. It's like how even though mathematicians ultimately do make errors, Mathematics has no replication crisis. However, most so-called 'hard' science are also full of knowledge that poorly models the world. Sadly, since most people who say this are also climate kooks, everyone will label you as a climate kook if you say this, though. But anyway, here's a fun thread https://twitter.com/MicrobiomDigest/status/1374046924551032835 https://twitter.com/MicrobiomDigest/status/13740469245510328... People constantly downvote me on HN for claiming that there are photoshopped papers out there. I'm not complaining about the downvotes. We make the culture of this place together and I can accept if I say things that you believe aren't right. But I am right. Edit since rate limited: oh, I didn't consider ML. Okay, consider my position reversed. Yeah, the empirical sciences do have reproducibility issues.
  - bildung 6y ago
    The ML field alone is full of unreproducable papers. Example: "“Probably 50%-75% of all papers are unreproducible. It’s sad, but it’s true,” another user wrote. “Think about it, most papers are ‘optimized’ to get into a conference. More often than not the authors know that a paper they’re trying to get into a conference isn’t very good! So they don’t have to worry about reproducibility because nobody will try to reproduce them.” https://bdtechtalks.com/2021/03/01/papers-without-code-machine-learning-reproducibility/ https://bdtechtalks.com/2021/03/01/papers-without-code-machi...
    - carschno 6y ago
      On the Papers without Code page, however, I see only 11 submitted papers, out of which 5 have been marked as resolved: https://airtable.com/shrWz8OF3uMZ8G4cY/tbl5ZzB7ahIui1EoD https://airtable.com/shrWz8OF3uMZ8G4cY/tbl5ZzB7ahIui1EoD This does not seem very good data to empirically underlay the claim of 50%-75% papers being irreproducible. I don't doubt that there are (too) many of them, but having some real data before making strong claims about numbers would be more credible.
- btilly 6y ago
  When I said that psychology was ground zero, I did not intend to imply that there wasn't a significant blast radius. That said, the blast radius is significantly less than you're describing. Most research in the hard sciences was not impacted at all. But that is in large part because they absorbed lessons about the difficulty and importance of replication many decades ago. And some of the lessons that they absorbed are being passed along in Feynman's speech.
lifeisstillgood 6y ago
I have this idea of a MOOP - Maasively Open Online Psychology. Basically it's - we have these constantly on monitoring devices, phones, watches, alexa, video - we can collect constant human behavioural data - and given the right regulatory framework (ie this is medical data treat it like that) we should see enormous commonalities (correlations) between millions of people. Psychology should then become a field of epidemiology- as opposed to limited behavioural studies in lab conditions. Maybe psychology just has not had enough in the field data?
- ChrisMarshallNY 6y ago
  I’m a fan of Psychohistory, myself. Has a strong Foundation in science, and is Seldon found to be incorrect... I’ll just get my coat...
  - lifeisstillgood 6y ago
    I feel that Asimov's Zeroth Law was always go for the pun. Several short stories I remember were basically just leading up to a bad pun. Coat not required :-)
    - Stratoscope 6y ago
      Now you're just telling me a Shah Guido G. story.
DoreenMichele 6y ago
Experiments involving human behavior are especially hard to design. Finding ways to figure out what people really do in real world conditions is quite a lot more challenging than most people want to believe. This is why I think all UBI experiments are bunk: The subjects know it is a limited-time experiment, not a new government policy, so they don't behave like they would if they believed this income was permanent for the rest of their life.
Pyramus 6y ago
It's easy and tempting to say things such as 'X is not science' or 'X is cargo cult science'. However, if you take your X to be large enough, like a whole discipline such as psychology, or computer science, or archaeology etc., the statement becomes a truism - not useful at best, misleading at worst. Let's take psychology as an example (use computer science in your mind if that's more familiar), it's a vast field spanning the whole spectrum from theoretical to experimental work. You start with something completely unscientific at one end and end up with the essence of the scientific method at the other.
- swebs 6y ago
  I think most people familiar with the field would rather classify Computer Science as a branch of mathematics instead. Its really an unfortunate name.

by btilly 6y ago

sadmann1 6y ago

If you think it's just psychology and not 99% of all publications in all domains including computer science you're quite optimistic

ta8645 6y ago
The lack of this being seriously addressed and remedied hurts the reputation of science in general. It also undermines the argument in areas of public policy around say Covid, where "listen to the science" has to overcome the perception that a lot of what passes for science... isn't.
- ramraj07 6y ago
  In Covid it was far more visibly clear this was and continues to happen. Everyone knows in their hearts that temp check is bullshit but that’s something everyone does even today. The about face on mask policy initially probably still plays a role in people’s perception and inability to follow rules today.
  - jjbinx007 6y ago
    > Everyone knows in their hearts that temp check is bullshit but that’s something everyone does even today. All our local visitor attractions (and I suspect it's the same UK wide) do temp checks on condition of entry. Pointing out the flaws with it is futile as it's the covid equivalent of security theatre. It ticks a box and makes it look like they are taking precautions. It reminds me of those snake oil bomb detectors that are still used in some parts of the world despite the fact that the 'inventors' have been jailed for selling the fraudulent devices: https://www.bbc.co.uk/news/uk-29459896 https://www.bbc.co.uk/news/uk-29459896
  - ACow_Adonis 6y ago
    what's bullshit about temp checks? if it's being sold as a silver bullet that stops all covid (or something else) , well sure, obviously that's bullshit. or is it the idea that it's OK to open everything if you just do temp checks, cause yeah, I can see how that would be bullshit too. But set a high enough threshold (to escape measurement error) and with accurate enough tools, it's one of the few public health things that seems somewhat evidence based: if you've got a verifiable fever, you probably shouldn't be out and mixing with others in confined spaces. hell, it probably made community health sense before COVID. obviously this doesn't stop non-fever, non symptomatic transmission, or people who dope themselves or their kids up to avoid detection, and it doesn't magically get rid of covid, but it picks out the low hanging fruit, and assuming properly calibrated equipment, seems like an obvious public health win to me (as I said, if you've got a fever, you should bloody well be going home/ denied entry to a workplace even in non-covid times).
    - ramraj07 6y ago
      By the time the 75% of patients get fever from Covid they are often fully aware that something’s not right and are probably not leaving home or have already gotten tested, only the crazy ones would still venture out in that stage. Importantly, the reason Covid spread this fast is because the disease seems to attain peak infectiousness before symptom onset and definitely before fever onset. This is why SARS was nipped in the bud with IR scanners while Covid wasn’t. You say that temperature checks don’t hurt but they most definitely do, most places where they do a check, especially in places like India, they think they’ve done enough and don’t enforce strict mask policies or distancing measures any further. If people are forced to stop using temp screening then these places will have to come up with alternative methods of comforting their audience they are taking measures so they would be forced to do actually effective screening methods like taking o2 measurements. So yeah it’s stupid to allow temp checks to pass for any theatre.
      - Angostura 6y ago
        >only the crazy ones would still venture out in that stage. Depending on location, that could still be a useful segment to exclude. People takkng their kids on a long-planned and long-promised visit to an attraction, who wake up feeling 'a bit under the weather' may be tempted to go anyway.
- paganel 6y ago
  Maybe we should try to detangle public policy from science for a while, it hurts both public policy and science.
  - KSteffensen 6y ago
    But shouldn't public policy be informed by (high-quality) science?
    - paganel 6y ago
      Why? Science (especially social sciences) is many, many times wrong, by definition (that's what science does, it gets less and less wrong until it reaches a certain epsilon of "less wrong" which it considers as the "truth"). This is not the fault of science because, again, errors/being wrong are included in its definition, so to speak, the problem is that those errors/wrongs, when applied to the life of real people, with real human lives, will most probably kill them or affect their lives negatively in very big ways, especially so if it's a concerted effort coming from the top. There are countless examples for that, the latest that comes to my mind is the pro-austerity policy imposed by the IMF on Greece, because that's what the science of economics was advising at the time, i.e. austerity. It turned out that that was a mistake (and IMF were quick to acknowledge it), but by that moment the damage had already been done. There's also the example of failed policies imposed by the likes of Robert McNamara and his men (they were mostly men) back in the 1960s, but I didn't catch those times in person, so I only know about those failures via links found on this website (and which I'm too lazy to search for right now).
      - bryanrasmussen 6y ago
        there's lots of public policy that is antagonistic to more grounded science than the social sciences, science that while it can be proven wrong at some point is unlikely to be and if it is proven wrong will probably be in small particulars rather than in the large. Evolutionary theory would be one obvious example.
      - _0ffh 6y ago
- youerbt 6y ago
  Well in case of public policy it's rather lack of trust in public policy. And in cases like Covid public policy should be ahead of science anyway, because there is no time to wait.
renewiltord 6y ago
The difference with CS is that there's no one to trust. Everything you need is right there. Other disciplines have people fake results. You can't fake a result in CS since people only really attack decidable problems, and you need no empirical results for any. It's like how even though mathematicians ultimately do make errors, Mathematics has no replication crisis. However, most so-called 'hard' science are also full of knowledge that poorly models the world. Sadly, since most people who say this are also climate kooks, everyone will label you as a climate kook if you say this, though. But anyway, here's a fun thread https://twitter.com/MicrobiomDigest/status/1374046924551032835 https://twitter.com/MicrobiomDigest/status/13740469245510328... People constantly downvote me on HN for claiming that there are photoshopped papers out there. I'm not complaining about the downvotes. We make the culture of this place together and I can accept if I say things that you believe aren't right. But I am right. Edit since rate limited: oh, I didn't consider ML. Okay, consider my position reversed. Yeah, the empirical sciences do have reproducibility issues.
- bildung 6y ago
  The ML field alone is full of unreproducable papers. Example: "“Probably 50%-75% of all papers are unreproducible. It’s sad, but it’s true,” another user wrote. “Think about it, most papers are ‘optimized’ to get into a conference. More often than not the authors know that a paper they’re trying to get into a conference isn’t very good! So they don’t have to worry about reproducibility because nobody will try to reproduce them.” https://bdtechtalks.com/2021/03/01/papers-without-code-machine-learning-reproducibility/ https://bdtechtalks.com/2021/03/01/papers-without-code-machi...
  - carschno 6y ago
    On the Papers without Code page, however, I see only 11 submitted papers, out of which 5 have been marked as resolved: https://airtable.com/shrWz8OF3uMZ8G4cY/tbl5ZzB7ahIui1EoD https://airtable.com/shrWz8OF3uMZ8G4cY/tbl5ZzB7ahIui1EoD This does not seem very good data to empirically underlay the claim of 50%-75% papers being irreproducible. I don't doubt that there are (too) many of them, but having some real data before making strong claims about numbers would be more credible.
btilly 6y ago
When I said that psychology was ground zero, I did not intend to imply that there wasn't a significant blast radius. That said, the blast radius is significantly less than you're describing. Most research in the hard sciences was not impacted at all. But that is in large part because they absorbed lessons about the difficulty and importance of replication many decades ago. And some of the lessons that they absorbed are being passed along in Feynman's speech.

lifeisstillgood 6y ago

I have this idea of a MOOP - Maasively Open Online Psychology. Basically it's - we have these constantly on monitoring devices, phones, watches, alexa, video - we can collect constant human behavioural data - and given the right regulatory framework (ie this is medical data treat it like that) we should see enormous commonalities (correlations) between millions of people. Psychology should then become a field of epidemiology- as opposed to limited behavioural studies in lab conditions. Maybe psychology just has not had enough in the field data?

ChrisMarshallNY 6y ago
I’m a fan of Psychohistory, myself. Has a strong Foundation in science, and is Seldon found to be incorrect... I’ll just get my coat...
- lifeisstillgood 6y ago
  I feel that Asimov's Zeroth Law was always go for the pun. Several short stories I remember were basically just leading up to a bad pun. Coat not required :-)
  - Stratoscope 6y ago
    Now you're just telling me a Shah Guido G. story.

DoreenMichele 6y ago

Experiments involving human behavior are especially hard to design. Finding ways to figure out what people really do in real world conditions is quite a lot more challenging than most people want to believe. This is why I think all UBI experiments are bunk: The subjects know it is a limited-time experiment, not a new government policy, so they don't behave like they would if they believed this income was permanent for the rest of their life.

Pyramus 6y ago

It's easy and tempting to say things such as 'X is not science' or 'X is cargo cult science'. However, if you take your X to be large enough, like a whole discipline such as psychology, or computer science, or archaeology etc., the statement becomes a truism - not useful at best, misleading at worst. Let's take psychology as an example (use computer science in your mind if that's more familiar), it's a vast field spanning the whole spectrum from theoretical to experimental work. You start with something completely unscientific at one end and end up with the essence of the scientific method at the other.

swebs 6y ago
I think most people familiar with the field would rather classify Computer Science as a branch of mathematics instead. Its really an unfortunate name.