3 ms·

There is a profound problem with the scientific process that I have been worrying about that rarely gets mentioned: The appeal and power of science is that you

There is a profound problem with the scientific process that I have been worrying about that rarely gets mentioned: The appeal and power of science is that you propose a model and then test it with evidence. With strong evidence you can validate the predictive efficacy of your model. However, the model is not "true", it is only predictive. Furthermore, it's only predictive to the extent that the evidence backs it up. The problem with this process is that it hangs entirely on the evidence. The models themselves do NOT emerge from the scientific process; they emerge from whatever - prejudice, confabulation, logic, supposition, etc. In other words, the original conjecture of a scientific model is merely a product of the pre-existing beliefs of the investigator. Coupling poor evidence to this sort of conjecture produces "science" that is simply supposition or bias with a veneer of credibility. Given the widespread abuse of methodology (p-hacking, poor understanding of statistics, small or unrepresentative sample sizes, not reporting negative experiments, bad experimental design) this means a lot of "science" is merely varnished prejudice.

dj-wonk 11y ago
I don't see this as a problem. Perhaps I define some concepts differently. As I see it, the scientific method is a good way to test a model against reality. It is less suited to be a "generative" process for making models. (There are other ways to test models too: internal consistency, interpretability, and so on.) Iteration, intuition, creativity, variation, intelligence, luck, guesswork, and more are what create theories. I'm not saying one could not using more "technical" or "rigorous" ways to construct models -- there are, of course, such methods. I'm just saying that "science" (in the Chalmers sense) is about mostly about testability of a model against evidence.
- astazangasta 11y ago
  >Iteration, intuition, creativity, variation, intelligence, luck, guesswork, and more are what create theories. These are all very positive. How about: racism, misogyny, bias, jealousy, and stubbornness. These, also, can feed into the creation of theories, and do. And, again, the point is that this process is only as good as the evidence; when the evidence is lacking, we're left with only the investigator's expertise to justify the model. That's not science.
  - dj-wonk 11y ago
    Excellent point. Yes, the sum total of human experience filters into model-making. People often construct theories that correspond with their preconceived biases. I would like to think that (a) "opening the lid" on how these models work and (b) testing them does a decent job of revealing and removing inaccurate or not-useful aspects. What do you think are some good ways to minimize the negative influences on models?
germinalphrase 11y ago
This feels particularly acute in the 'soft sciences' (e.g. Educational research) where the bar for truth-y results is even that much more diminished. I look forward to the day that educators use a common platform for planning/performance/assessment allowing for large-scale data analysis... But that's not even on the radar for in-person educational environments.

dj-wonk 11y ago

I don't see this as a problem. Perhaps I define some concepts differently. As I see it, the scientific method is a good way to test a model against reality. It is less suited to be a "generative" process for making models. (There are other ways to test models too: internal consistency, interpretability, and so on.) Iteration, intuition, creativity, variation, intelligence, luck, guesswork, and more are what create theories. I'm not saying one could not using more "technical" or "rigorous" ways to construct models -- there are, of course, such methods. I'm just saying that "science" (in the Chalmers sense) is about mostly about testability of a model against evidence.

astazangasta 11y ago
>Iteration, intuition, creativity, variation, intelligence, luck, guesswork, and more are what create theories. These are all very positive. How about: racism, misogyny, bias, jealousy, and stubbornness. These, also, can feed into the creation of theories, and do. And, again, the point is that this process is only as good as the evidence; when the evidence is lacking, we're left with only the investigator's expertise to justify the model. That's not science.
- dj-wonk 11y ago
  Excellent point. Yes, the sum total of human experience filters into model-making. People often construct theories that correspond with their preconceived biases. I would like to think that (a) "opening the lid" on how these models work and (b) testing them does a decent job of revealing and removing inaccurate or not-useful aspects. What do you think are some good ways to minimize the negative influences on models?

germinalphrase 11y ago

This feels particularly acute in the 'soft sciences' (e.g. Educational research) where the bar for truth-y results is even that much more diminished. I look forward to the day that educators use a common platform for planning/performance/assessment allowing for large-scale data analysis... But that's not even on the radar for in-person educational environments.