5 ms·

The problem with “is this true?” Investigations is that there are tons of potential pitfalls and a combination of experimental limitations, poor academic incent

The problem with “is this true?” Investigations is that there are tons of potential pitfalls and a combination of experimental limitations, poor academic incentives, and insufficient statistical and scientific training make this very hard to do well. I think it’s bad science, that will one day retrospectively be viewed as invalid scientific method. I think the way to combat this is to avoid studies that solely attempt to demonstrate an effect, but to complement that part of an investigation with additional studies that demonstrate a plausible explanation. This reduces the risk of false positives and increases our confidence in the result. The additional studies could take the form (for example) of deliberately intervening such that the effect should be manipulated by the intervention in a predictable and measurable way. Essentially, come at the hypothesis “from another angle”.

deleted 9y ago
[deleted]
jeffdavis 9y ago
I somewhat agree, but "is it true?" is all that science can answer. A mechanism is really just a narrower example without irrelevant distractions or confounding variables. Kind of like the difference between the bug reports: "run this complex workload on this specific machine and it will orobably crash after a few hours" versus "run these three commands when the cache is cold and it crashes".
- fleitz 9y ago
  Actually all science can really answer is 'is it false?'.
- jjgreen 9y ago
  I find it odd that the sibling reply by fleitz (all that science can say is "is it false"), which just a precis of Popperian philosophy of science, has been marked dead. Not a view that I share, but seems uncontroversial.
  - pbhjpbhj 9y ago
    Agreed, indeed my immediate response to that comment was "that's not how science works, it's falsificationist". What is your view on science, do you think it can create - and verify - a true result? Whither axiomatic underpinning in such a view?
    - jjgreen 9y ago
      Well thanks for asking. I think it's clear that science gets us truth, but that the mechanism for that complex social activity can't be reduced to a few axioms or principles; Feyerabend gives a detailed description [1] of how bad a scientist Galileo was (as judged by modern theories of what science is), yet he was right, and he advanced truth. [1] Paul Feyerabend, Against Method (1975)
      - pbhjpbhj 9y ago
        And thanks for responding ... Heliocentrism isn't truth, it's opinion [that's my opinion, lol!!]. Einstein's relativity makes heliocentrism false in demanding that there is no aether and so no special points in space. Science provides our understanding of a best current description of reality - the whole point of scientific endeavour is to demonstrate we were wrong and that our understanding/description was flawed. Moreover it relies on maths, which is provably unable to verify it's own consistency and completeness. Furthermore it's an axiomatic system that produces a probabilistic framework that gives us little to hang our conception of reality on [what I'm saying is our perception is not particularly informed by physical reality, we have to subdue our instinctive perceptions to work with what amounts to our best, most consistent, physics view of reality]. Science gives us as much truth as we can see in a well lit test-tube.
        senderista 9y ago
        Actually GR makes both geocentrism and heliocentrism "true". One is free to pick whatever coordinate system is most convenient to solve a given problem.
dumbneurologist 9y ago
there are tons of potential pitfalls and a combination of experimental limitations you provided some, which are at least partially addressed by the study design. (I don't think it's your point, but let me agree that "the scientist can cheat" is always true). poor academic incentives, this is addressed because it is double-blind insufficient statistical and scientific training this is (partially) addressed by peer review, and the availability of the data. you can't make this general criticism of the paper at hand: find the place where the design or statistics used were in error. I think it’s bad science, that will one day retrospectively be viewed as invalid scientific method. that's a claim without any evidence to back it p. I think the way to combat this is to avoid studies that solely attempt to demonstrate an effect, but to complement that part of an investigation with additional studies that demonstrate a plausible explanation. This reduces the risk of false positives and increases our confidence in the result. If you have a double-blind, placebo-controlled, randomized clinical trial then what can the mechanism possibly add? Is an effective medication less effective because we don't understand how it works? If you want an amazing example, read about the mechanism of action of general anesthesia: we have many ideas about how it works, but nobody really knows. Nobody gets caught up by this when their appendix ruptures. I hope, for your sake, that you will lower your standards in the event that you need surgery. The additional studies could take the form (for example) of deliberately intervening such that the effect should be manipulated by the intervention in a predictable and measurable way. Essentially, come at the hypothesis “from another angle”. If you have a properly designed and executed randomized, double blind, placebo controlled trial, then you have measured the effect of the intervention, and only the intervention. You can repeat it in order to gain confidence, but you have to decide when the diminishing returns of "more confidence" are not worth your study dollar. And for curcurmin, which is cheap and has low side effects... it's probably not that far away. edit: fixed formatting
- electrograv 9y ago
  >> I think it’s bad science, that will one day retrospectively be viewed as invalid scientific method. > that's a claim without any evidence to back it [u]p. You are correct. The irony is rich here. I always welcome a good argument (whether in dispute or affirmation of an issue), but just saying "I think <XYZ>" doesn't make it so. Reading through this thread, all I see is the common HN Default Skepticism / Negativity -- but more so; skepticism with less scientific rigor than the work being criticized.

deleted 9y ago

[deleted]

jeffdavis 9y ago

I somewhat agree, but "is it true?" is all that science can answer. A mechanism is really just a narrower example without irrelevant distractions or confounding variables. Kind of like the difference between the bug reports: "run this complex workload on this specific machine and it will orobably crash after a few hours" versus "run these three commands when the cache is cold and it crashes".

fleitz 9y ago
Actually all science can really answer is 'is it false?'.
jjgreen 9y ago
I find it odd that the sibling reply by fleitz (all that science can say is "is it false"), which just a precis of Popperian philosophy of science, has been marked dead. Not a view that I share, but seems uncontroversial.
- pbhjpbhj 9y ago
  Agreed, indeed my immediate response to that comment was "that's not how science works, it's falsificationist". What is your view on science, do you think it can create - and verify - a true result? Whither axiomatic underpinning in such a view?
  - jjgreen 9y ago
    Well thanks for asking. I think it's clear that science gets us truth, but that the mechanism for that complex social activity can't be reduced to a few axioms or principles; Feyerabend gives a detailed description [1] of how bad a scientist Galileo was (as judged by modern theories of what science is), yet he was right, and he advanced truth. [1] Paul Feyerabend, Against Method (1975)
    - pbhjpbhj 9y ago
      And thanks for responding ... Heliocentrism isn't truth, it's opinion [that's my opinion, lol!!]. Einstein's relativity makes heliocentrism false in demanding that there is no aether and so no special points in space. Science provides our understanding of a best current description of reality - the whole point of scientific endeavour is to demonstrate we were wrong and that our understanding/description was flawed. Moreover it relies on maths, which is provably unable to verify it's own consistency and completeness. Furthermore it's an axiomatic system that produces a probabilistic framework that gives us little to hang our conception of reality on [what I'm saying is our perception is not particularly informed by physical reality, we have to subdue our instinctive perceptions to work with what amounts to our best, most consistent, physics view of reality]. Science gives us as much truth as we can see in a well lit test-tube.
      - senderista 9y ago
        Actually GR makes both geocentrism and heliocentrism "true". One is free to pick whatever coordinate system is most convenient to solve a given problem.

dumbneurologist 9y ago

there are tons of potential pitfalls and a combination of experimental limitations you provided some, which are at least partially addressed by the study design. (I don't think it's your point, but let me agree that "the scientist can cheat" is always true). poor academic incentives, this is addressed because it is double-blind insufficient statistical and scientific training this is (partially) addressed by peer review, and the availability of the data. you can't make this general criticism of the paper at hand: find the place where the design or statistics used were in error. I think it’s bad science, that will one day retrospectively be viewed as invalid scientific method. that's a claim without any evidence to back it p. I think the way to combat this is to avoid studies that solely attempt to demonstrate an effect, but to complement that part of an investigation with additional studies that demonstrate a plausible explanation. This reduces the risk of false positives and increases our confidence in the result. If you have a double-blind, placebo-controlled, randomized clinical trial then what can the mechanism possibly add? Is an effective medication less effective because we don't understand how it works? If you want an amazing example, read about the mechanism of action of general anesthesia: we have many ideas about how it works, but nobody really knows. Nobody gets caught up by this when their appendix ruptures. I hope, for your sake, that you will lower your standards in the event that you need surgery. The additional studies could take the form (for example) of deliberately intervening such that the effect should be manipulated by the intervention in a predictable and measurable way. Essentially, come at the hypothesis “from another angle”. If you have a properly designed and executed randomized, double blind, placebo controlled trial, then you have measured the effect of the intervention, and only the intervention. You can repeat it in order to gain confidence, but you have to decide when the diminishing returns of "more confidence" are not worth your study dollar. And for curcurmin, which is cheap and has low side effects... it's probably not that far away. edit: fixed formatting

electrograv 9y ago
>> I think it’s bad science, that will one day retrospectively be viewed as invalid scientific method. > that's a claim without any evidence to back it [u]p. You are correct. The irony is rich here. I always welcome a good argument (whether in dispute or affirmation of an issue), but just saying "I think <XYZ>" doesn't make it so. Reading through this thread, all I see is the common HN Default Skepticism / Negativity -- but more so; skepticism with less scientific rigor than the work being criticized.