6 ms·

Hardly new. Feynman was criticizing psych research for this sort of thing - inadequate controls - half a century ago in his talk on Cargo Cult Science. http://c

Hardly new. Feynman was criticizing psych research for this sort of thing - inadequate controls - half a century ago in his talk on Cargo Cult Science. http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/CargoCult.htm

mayank 9y ago
Inadequate controls = attempting to measure fine effects (gender normalized animal behavior) using crude methods (rat study).
- colehasson 9y ago
  Accuracy or precision. Pick one?
  - marcosdumay 9y ago
    Well, why should anybody not pick accuracy every time?
    - nerdponx 9y ago
      Because, in the extreme, accuracy without precision doesn't tell you a damn thing.
      - marcosdumay 9y ago
        Just like precision without accuracy, but if you get accuracy you know you have a problem, but with only precision you are hopelessly misled. If it was a strict trade-off, I would agree that the answer was somewhere on the middle. But it isn't. Most of the things that give you precision will also give you accuracy. The few things that let you exchange one for the other look too much like cheating, and seem to always have much greater impacts on accuracy than on precision.
        kbutler 9y ago
        > Most of the things that give you precision will also give you accuracy. This is absolutely false - digital clocks are a common example. Precise to the second (or millisecond, or more!) but only as accurate as their setting. Calculations also give precision without accuracy, like my package of tortilla chips - 13oz (368.5g) They took a measurement or specification precise to the ounce, multiplied by 28.3495, and got 368.5. Accuracy and precision are independent variables, and the utility of either is usually limited to the order of magnitude of the other. With either "accurate to the second with a precision of milliseconds", or "accurate to the millisecond with precision to the second", you should only report a measurement to the second (or possibly a more precise measurement with an error range).
    - kbutler 9y ago
      Where is the closest gas station? "Within the Solar System".
  - felipemnoa 9y ago
    >>Accuracy or precision Do they have different meaning to you? They are the same thing unless there is some scientific meaning that makes them different of which I'm not aware. i.e. Theory in Science means a different thing than what it means to a layman person.
    - gnajob 9y ago
      https://en.wikipedia.org/wiki/Accuracy_and_precision https://en.wikipedia.org/wiki/Accuracy_and_precision TL;DR: https://phidgets.files.wordpress.com/2014/05/precision-accuracy.png https://phidgets.files.wordpress.com/2014/05/precision-accur...
    - macintux 9y ago
      I admit I'm only skimming the discussion so I may be missing the point of your question, but a common analogy to the difference: If you're target shooting, and all of your shots go through the same hole: that's precision. If that hole is several inches away from the bullseye, it's precise but not accurate.
    - jodoherty 9y ago
      Precision is how fine a measurement is (i.e. being able to measure temperature to thousandths of degrees versus tenths), whereas accuracy is a measure of how closely your results much the objective truth. So you may have a very precise thermometer that measures to thousandths of degrees but it may not be accurate because it's calibrated such that it always reads exactly two degrees higher than it should. Meanwhile a thermometer that can only measure to a tenth of a degree is more accurate and less precise than the one just mentioned if the shown figures reflect the true temperature. In common usage, these words mean the same thing, but scientists often give words specific meanings in order to make their research and ideas less ambiguous. Definitions are even more important in non-scientific fields such as philosophy or math where meaningful reasoning of abstract structures and ideas would be next to impossible without giving them concrete definitions and stating your assumptions. Remember that definitions are arbitrary, so to understand an author's argument or idea, you must seek out the author's definitions. Hopefully that clarifies some of the discussion going on in the comments here.
raverbashing 9y ago
And what happens when you have more variables than you can control? It's naive to think all of them can be controlled, even in simple animals as mice
- SomeStupidPoint 9y ago
  Then it's outright fraud to present your results as reliable when there's radically uncontrolled variables that dominate the results.
  - sideshowb 9y ago
    It would only be fraud if done with deliberate intent to deceive.
  - Cthulhu_ 9y ago
    Just make sure to note the variables if you can list them. The main problem is media outlets taking the results as fact.
  - nonbel 9y ago
    This is also naive. In that case you won't be able to publish your results in prestigious journals and your career will die out. That said, you do learn not to act 100% confident. You need to say "This suggests that" instead of "This proves that", etc. Anyway, the solution to reliable data is simple: Have multiple competing groups try to repeat the work as closely as possible. Now that you have something to hang your hat on, you come up with some theory/model to explain the phenomenon. Even better, come up with multiple competing such models. From these you deduce some precise prediction that is checked against future data. All of this was well known a few generations ago...
    - SomeStupidPoint 9y ago
      > This is also naive. In that case you won't be able to publish your results in prestigious journals and your career will die out. Charlatan fortunetellers aren't owed careers as scientists -- if they can't produce reliable, predictive models because they utterly fail to control experiments,then they should have their careers destroyed. We've seen enough damage from corporate leaders and politicians relying on guides to reading chicken entrails masquerading as science. It's time to stop the fraud.

mayank 9y ago

Inadequate controls = attempting to measure fine effects (gender normalized animal behavior) using crude methods (rat study).

colehasson 9y ago
Accuracy or precision. Pick one?
- marcosdumay 9y ago
  Well, why should anybody not pick accuracy every time?
  - nerdponx 9y ago
    Because, in the extreme, accuracy without precision doesn't tell you a damn thing.
    - marcosdumay 9y ago
      Just like precision without accuracy, but if you get accuracy you know you have a problem, but with only precision you are hopelessly misled. If it was a strict trade-off, I would agree that the answer was somewhere on the middle. But it isn't. Most of the things that give you precision will also give you accuracy. The few things that let you exchange one for the other look too much like cheating, and seem to always have much greater impacts on accuracy than on precision.
      - kbutler 9y ago
        > Most of the things that give you precision will also give you accuracy. This is absolutely false - digital clocks are a common example. Precise to the second (or millisecond, or more!) but only as accurate as their setting. Calculations also give precision without accuracy, like my package of tortilla chips - 13oz (368.5g) They took a measurement or specification precise to the ounce, multiplied by 28.3495, and got 368.5. Accuracy and precision are independent variables, and the utility of either is usually limited to the order of magnitude of the other. With either "accurate to the second with a precision of milliseconds", or "accurate to the millisecond with precision to the second", you should only report a measurement to the second (or possibly a more precise measurement with an error range).
  - kbutler 9y ago
    Where is the closest gas station? "Within the Solar System".
- felipemnoa 9y ago
  >>Accuracy or precision Do they have different meaning to you? They are the same thing unless there is some scientific meaning that makes them different of which I'm not aware. i.e. Theory in Science means a different thing than what it means to a layman person.
  - gnajob 9y ago
    https://en.wikipedia.org/wiki/Accuracy_and_precision https://en.wikipedia.org/wiki/Accuracy_and_precision TL;DR: https://phidgets.files.wordpress.com/2014/05/precision-accuracy.png https://phidgets.files.wordpress.com/2014/05/precision-accur...
  - macintux 9y ago
    I admit I'm only skimming the discussion so I may be missing the point of your question, but a common analogy to the difference: If you're target shooting, and all of your shots go through the same hole: that's precision. If that hole is several inches away from the bullseye, it's precise but not accurate.
  - jodoherty 9y ago
    Precision is how fine a measurement is (i.e. being able to measure temperature to thousandths of degrees versus tenths), whereas accuracy is a measure of how closely your results much the objective truth. So you may have a very precise thermometer that measures to thousandths of degrees but it may not be accurate because it's calibrated such that it always reads exactly two degrees higher than it should. Meanwhile a thermometer that can only measure to a tenth of a degree is more accurate and less precise than the one just mentioned if the shown figures reflect the true temperature. In common usage, these words mean the same thing, but scientists often give words specific meanings in order to make their research and ideas less ambiguous. Definitions are even more important in non-scientific fields such as philosophy or math where meaningful reasoning of abstract structures and ideas would be next to impossible without giving them concrete definitions and stating your assumptions. Remember that definitions are arbitrary, so to understand an author's argument or idea, you must seek out the author's definitions. Hopefully that clarifies some of the discussion going on in the comments here.

raverbashing 9y ago

And what happens when you have more variables than you can control? It's naive to think all of them can be controlled, even in simple animals as mice

SomeStupidPoint 9y ago
Then it's outright fraud to present your results as reliable when there's radically uncontrolled variables that dominate the results.
- sideshowb 9y ago
  It would only be fraud if done with deliberate intent to deceive.
- Cthulhu_ 9y ago
  Just make sure to note the variables if you can list them. The main problem is media outlets taking the results as fact.
- nonbel 9y ago
  This is also naive. In that case you won't be able to publish your results in prestigious journals and your career will die out. That said, you do learn not to act 100% confident. You need to say "This suggests that" instead of "This proves that", etc. Anyway, the solution to reliable data is simple: Have multiple competing groups try to repeat the work as closely as possible. Now that you have something to hang your hat on, you come up with some theory/model to explain the phenomenon. Even better, come up with multiple competing such models. From these you deduce some precise prediction that is checked against future data. All of this was well known a few generations ago...
  - SomeStupidPoint 9y ago
    > This is also naive. In that case you won't be able to publish your results in prestigious journals and your career will die out. Charlatan fortunetellers aren't owed careers as scientists -- if they can't produce reliable, predictive models because they utterly fail to control experiments,then they should have their careers destroyed. We've seen enough damage from corporate leaders and politicians relying on guides to reading chicken entrails masquerading as science. It's time to stop the fraud.