4 ms·

If what you've read here concerns you to the point of throwing out the field, you're going to be immensely disappointed by every other scientific field, as well

If what you've read here concerns you to the point of throwing out the field, you're going to be immensely disappointed by every other scientific field, as well. [0] I'm not an expert, but my understanding is that this article could have been written about literally any field of science, with a depressingly few nouns changed. [1] This is what people mean when they say, "science is messy". This is how the sausage gets made, and if you're cool throwing out psychology as a field, you're probably also going to have to be cool throwing out a lot of other fields. [2] There are efforts being made to clean up how science is conducted, called "metascience", but it's a relatively new field and there's still a lot that hasn't been investigated yet. [3] Basically, if you've ever thought of yourself as "pro science", you've been supporting what happened in this article, whether you realized it or not. [0] https://www.nature.com/articles/533452a https://www.nature.com/articles/533452a [1] https://www.scientificamerican.com/video/is-there-a-reproducibility-crisis-in-science/ https://www.scientificamerican.com/video/is-there-a-reproduc... -- a slightly more accessible version of [0] [2] https://direct.mit.edu/posc/article/28/4/482/97500/Mess-in-Science-and-Wicked-Problems https://direct.mit.edu/posc/article/28/4/482/97500/Mess-in-S... [3] https://en.wikipedia.org/wiki/Metascience https://en.wikipedia.org/wiki/Metascience

fhood 5y ago
I think we can maybe inject a little bit of extra nuance here. Not every field is the same, they vary massively in size, level of external influence, and complexity in subject matter. Psychology suffers more than most. It's a large field, so competition to get noticed is likely high. There is money to be made, as discussed in the article, so it struggles with influences outside of pure scientific inquiry, and they have no chance of describing their theories in terms of pure component mathematics. Contrast that with ocean and atmospheric sciences. It is a tiny and very insular field, so the rat race works very differently. This also protects them from outside influences, and a lot of the major underlying physical and biological processes involved can be modeled with mathematics that have been demonstrated to be predictive. It's just easier for some fields to reliably produce higher quality work. I think this is a case where blanket skepticism is just lazy. Every field suffers some level of a reproducibility issue, but some fields are better able to self select for a variety of reasons.
ModernMech 5y ago
Thank you for saying this. I mean, Computer Science is up there in the junk science department. Coming from a phsyics background, it's hard to even call most of what happens in CS "science". Go to any CS conference and you'll see a whole bunch of papers doing micro benchmarks, saying that their method is X% faster than all the others, where X% is usually some very low percentage (and of course no errors bars are visible on any plots). You talk with the guy presenting the research and you find out that half the benchmarks were done on one machine, while the other half were done on another with different specs. Well, okay, now why can I trust the benchmarks? And how many other papers are doing this? And is your result even meaningful anymore? It's so depressing. And don't even get me started on the AI/ML field. My god, talk about art over science.
- bonoboTP 5y ago
  A lot of AI/ML work is sloppy but well known methods work reliably, textbook descriptions make sense. That there's a long tail of shitty student papers doesn't invalidate the field. Year after year methods genuinely get better. Whatever process produces this progress, it produces this progress.
- jeltz 5y ago
  Also it is very common for these benchmarks to conveniently leave out the fastest current known methods for unclear reasons.
pdonis 5y ago
> you're going to be immensely disappointed by every other scientific field, as well. Not every other field. We do have scientific theories that are reliable. But the reason they are reliable is that they are nailed down by extensive testing in controlled experiments that have confirmed their predictions to many decimal places. (I am speaking, of course, of General Relativity and the Standard Model of particle physics.) So we do know what actual reliable science looks like. The problem, of course, is that there is not a lot of other science that actually looks like that, but it still gets called "science" anyway, and treated as though it is reliable even though it's not.
- TameAntelope 5y ago
  I said "every" because even some of the best work science has done, General Relativity and the Standard Model, were to my understanding an incompatible set of theories. I did a bit of brief Googling, and it looks like some of that has been resolved and it isn't seen as totally incompatible anymore, but even that's not locked down as of yet. The Wikipedia page is kind of a mess, but I'll link it here for others to save you a few clicks: https://en.wikipedia.org/wiki/Physics_beyond_the_Standard_Model https://en.wikipedia.org/wiki/Physics_beyond_the_Standard_Mo... Of note, it looks like there's a debate ongoing even within the citations about compatibility/non-compatibility, which I've never seen before on a Wikipedia page (I've done some light editing to separate out the relevant quotes from their related citations to improve readability): >> Sushkov, A. O.; Kim, W. J.; Dalvit, D. A. R.; Lamoreaux, S. K. (2011). "New Experimental Limits on Non-Newtonian Forces in the Micrometer Range". Physical Review Letters. 107 (17): 171101. arXiv:1108.2547. Bibcode:2011PhRvL.107q1101S. doi:10.1103/PhysRevLett.107.171101. PMID 22107498. S2CID 46596924. > "It is remarkable that two of the greatest successes of 20th century physics, general relativity and the standard model, appear to be fundamentally incompatible." >> But see also Donoghue, John F. (2012). "The effective field theory treatment of quantum gravity". AIP Conference Proceedings. 1473 (1): 73. arXiv:1209.3511. Bibcode:2012AIPC.1483...73D. doi:10.1063/1.4756964. S2CID 119238707. > "One can find thousands of statements in the literature to the effect that "general relativity and quantum mechanics are incompatible". These are completely outdated and no longer relevant. Effective field theory shows that general relativity and quantum mechanics work together perfectly normally over a range of scales and curvatures, including those relevant for the world that we see around us. However, effective field theories are only valid over some range of scales. General relativity certainly does have problematic issues at extreme scales. There are important problems which the effective field theory does not solve because they are beyond its range of validity. However, this means that the issue of quantum gravity is not what we thought it to be. Rather than a fundamental incompatibility of quantum mechanics and gravity, we are in the more familiar situation of needing a more complete theory beyond the range of their combined applicability. The usual marriage of general relativity and quantum mechanics is fine at ordinary energies, but we now seek to uncover the modifications that must be present in more extreme conditions. This is the modern view of the problem of quantum gravity, and it represents progress over the outdated view of the past."

fhood 5y ago

I think we can maybe inject a little bit of extra nuance here. Not every field is the same, they vary massively in size, level of external influence, and complexity in subject matter. Psychology suffers more than most. It's a large field, so competition to get noticed is likely high. There is money to be made, as discussed in the article, so it struggles with influences outside of pure scientific inquiry, and they have no chance of describing their theories in terms of pure component mathematics. Contrast that with ocean and atmospheric sciences. It is a tiny and very insular field, so the rat race works very differently. This also protects them from outside influences, and a lot of the major underlying physical and biological processes involved can be modeled with mathematics that have been demonstrated to be predictive. It's just easier for some fields to reliably produce higher quality work. I think this is a case where blanket skepticism is just lazy. Every field suffers some level of a reproducibility issue, but some fields are better able to self select for a variety of reasons.

ModernMech 5y ago

Thank you for saying this. I mean, Computer Science is up there in the junk science department. Coming from a phsyics background, it's hard to even call most of what happens in CS "science". Go to any CS conference and you'll see a whole bunch of papers doing micro benchmarks, saying that their method is X% faster than all the others, where X% is usually some very low percentage (and of course no errors bars are visible on any plots). You talk with the guy presenting the research and you find out that half the benchmarks were done on one machine, while the other half were done on another with different specs. Well, okay, now why can I trust the benchmarks? And how many other papers are doing this? And is your result even meaningful anymore? It's so depressing. And don't even get me started on the AI/ML field. My god, talk about art over science.

bonoboTP 5y ago
A lot of AI/ML work is sloppy but well known methods work reliably, textbook descriptions make sense. That there's a long tail of shitty student papers doesn't invalidate the field. Year after year methods genuinely get better. Whatever process produces this progress, it produces this progress.
jeltz 5y ago
Also it is very common for these benchmarks to conveniently leave out the fastest current known methods for unclear reasons.

pdonis 5y ago

> you're going to be immensely disappointed by every other scientific field, as well. Not every other field. We do have scientific theories that are reliable. But the reason they are reliable is that they are nailed down by extensive testing in controlled experiments that have confirmed their predictions to many decimal places. (I am speaking, of course, of General Relativity and the Standard Model of particle physics.) So we do know what actual reliable science looks like. The problem, of course, is that there is not a lot of other science that actually looks like that, but it still gets called "science" anyway, and treated as though it is reliable even though it's not.

TameAntelope 5y ago
I said "every" because even some of the best work science has done, General Relativity and the Standard Model, were to my understanding an incompatible set of theories. I did a bit of brief Googling, and it looks like some of that has been resolved and it isn't seen as totally incompatible anymore, but even that's not locked down as of yet. The Wikipedia page is kind of a mess, but I'll link it here for others to save you a few clicks: https://en.wikipedia.org/wiki/Physics_beyond_the_Standard_Model https://en.wikipedia.org/wiki/Physics_beyond_the_Standard_Mo... Of note, it looks like there's a debate ongoing even within the citations about compatibility/non-compatibility, which I've never seen before on a Wikipedia page (I've done some light editing to separate out the relevant quotes from their related citations to improve readability): >> Sushkov, A. O.; Kim, W. J.; Dalvit, D. A. R.; Lamoreaux, S. K. (2011). "New Experimental Limits on Non-Newtonian Forces in the Micrometer Range". Physical Review Letters. 107 (17): 171101. arXiv:1108.2547. Bibcode:2011PhRvL.107q1101S. doi:10.1103/PhysRevLett.107.171101. PMID 22107498. S2CID 46596924. > "It is remarkable that two of the greatest successes of 20th century physics, general relativity and the standard model, appear to be fundamentally incompatible." >> But see also Donoghue, John F. (2012). "The effective field theory treatment of quantum gravity". AIP Conference Proceedings. 1473 (1): 73. arXiv:1209.3511. Bibcode:2012AIPC.1483...73D. doi:10.1063/1.4756964. S2CID 119238707. > "One can find thousands of statements in the literature to the effect that "general relativity and quantum mechanics are incompatible". These are completely outdated and no longer relevant. Effective field theory shows that general relativity and quantum mechanics work together perfectly normally over a range of scales and curvatures, including those relevant for the world that we see around us. However, effective field theories are only valid over some range of scales. General relativity certainly does have problematic issues at extreme scales. There are important problems which the effective field theory does not solve because they are beyond its range of validity. However, this means that the issue of quantum gravity is not what we thought it to be. Rather than a fundamental incompatibility of quantum mechanics and gravity, we are in the more familiar situation of needing a more complete theory beyond the range of their combined applicability. The usual marriage of general relativity and quantum mechanics is fine at ordinary energies, but we now seek to uncover the modifications that must be present in more extreme conditions. This is the modern view of the problem of quantum gravity, and it represents progress over the outdated view of the past."