7 ms·

Nice, Feynman also described this in cargo cult science:(http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/Cargo

Nice, Feynman also described this in cargo cult science:(http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/CargoCult.htm) > We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops and got an answer which we now know not to be quite right. It’s a little bit off, because he had the incorrect value for the viscosity of air. It’s interesting to look at the history of measurements of the charge of the electron, after Millikan. If you plot them as a function of time, you find that one is a little bigger than Millikan’s, and the next one’s a little bit bigger than that, and the next one’s a little bit bigger than that, until finally they settle down to a number which is higher. > Why didn’t they discover that the new number was higher right away? It’s a thing that scientists are ashamed of—this history—because it’s apparent that people did things like this: When they got a number that was too high above Millikan’s, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number closer to Millikan’s value they didn’t look so hard. And so they eliminated the numbers that were too far off, and did other things like that. We’ve learned those tricks nowadays, and now we don’t have that kind of a disease.

afthonos 7y ago
> We’ve learned those tricks nowadays, and now we don’t have that kind of a disease. That’s very optimistic. If nowhere else, polling has a huge herding problem, where outliers are dropped when they are too far off the consensus.
- zamalek 7y ago
  I'm pretty certain that's satire, or would be satire if Feynman were around to see the state of modern science (interesting things off-the-beaten-path were being done in his day).
chrisco255 7y ago
What evidence do you have that this sort of thing is not occurring nowadays?
- leoc 7y ago
  Well, the GP is quoting Feynman, so I'm not sure he's endorsing that part of the quotation. FWIW I think it's another healthy warning sign suggesting not to receive Feynman's pronouncements about scientific method too uncritically. However it may also be that the situation is now actually worse than when Feynman wrote this.
spats1990 7y ago
>One example of the principle is this: If you’ve made up your mind to test a theory, or you want to explain some idea, you should always decide to publish it whichever way it comes out. If we only publish results of a certain kind, we can make the argument look good. We must publish both kinds of result. Isn't non-publication of studies/research (for various reasons) a massive problem in a lot of different disciplines now? [1], for example. [1]https://pediatrics.aappublications.org/content/138/3/e20160223 https://pediatrics.aappublications.org/content/138/3/e201602...
- bonoboTP 7y ago
  Mainly because reviewers won't accept uninteresting results. In many fields it's hard to differentiate a negative result from just a clumsy researcher. In computer science a negative result usually generates a flood of questions like "okay, but why would you do it like that, couldn't you try this or that etc." and the conclusion is that the researcher just didn't manage to get it to work because they didn't use the right techniques. Essentially you'd have to argue that the method you tried should work, based on what we already know, but surprisingly it doesn't work. This type of argument is very hard to carry through. The opposite is much more publication friendly: "nobody would have thought that this could ever work like this, but we now show that this novel idea does work very well". That's the type of thing for award-winning research.
  - lallysingh 7y ago
    Publication -- somewhere -- of interesting negative results would be invaluable specifically for the discussion of how the experiment went wrong. Experiment design is tricky and it's not something you get a lot of exposure to except through the experience of others you interact with.
    - sifar 7y ago
      Eliezer Yudkowsky talks about this in Inadequate Equiibria [1] when he analyzes academic research. short answer - here you need researchers and grant makers to simultaneously decide to do what is not in the best interest for them. [1] https://equilibriabook.com/an-equilibrium-of-no-free-energy/ https://equilibriabook.com/an-equilibrium-of-no-free-energy/
foldr 7y ago
>We’ve learned those tricks nowadays, and now we don’t have that kind of a disease. I'm sure the people doing those experiments also shared the conviction that they were smarter and more objective than people in past ages.
- jstummbillig 7y ago
  But we are getting better, right? Despite undoubtedly repeating a lot mistakes (and often times being foolishly ignorant of that), some collective knowledge is surely being retained, otherwise we would not make progress in basically all sciences.
  - foldr 7y ago
    The fact that we're making progress in acquiring knowledge doesn't necessarily entail that we're getting better at being objective or unbiased. I see no reason to think that we are. (Nor am I particularly convinced that objectivity and lack of bias are major factors driving scientific progress, but that's probably another conversation.)
    - SiempreViernes 7y ago
      The very experiences that Feynman recounts have lead to a practice of doing "blind" experiments in particle physics and other places. You simply do your analysis with some extra, unknown, factor added and once you think you done the best you can, the blinding is removed and you check to see what you measured.
      - foldr 7y ago
        Sure, but who knows if this has been beneficial to scientific progress or not? It's not like we can wind back time and do a comparative experiment. edit To expand on this a bit, as it might sound flippant. People often talk about "bias" as if it's something that obviously ought to be eliminated. But in fact you can't get anywhere without biases. We only have the time and resources to explore a tiny portion of the hypothesis space in most domains.
dr_dshiv 7y ago
Isn't that also an appropriately Bayesian approach to new evidence?
- marcosdumay 7y ago
  No, it isn't. The appropriate approach is to accept the evidence, and correct your priors based on it, even if it's not good enough to believe. It's not to fiddle with the evidence until it's something you believe is correct. Of course, it's much easier said than done. I don't think any group of scientists is safe from repeating this.
- SatvikBeri 7y ago
  The correct approach would be to say something like "Previous experiments reported X, we observed X + Y, therefore the real answer is probably around X + Y/10." But it's very important to state the observation of X + Y, not X + Y/10, otherwise you'll take much too long to update to the correct value!
- darkmighty 7y ago
  Being a good Bayesian would entail taking the evidence as-is, only adjusting your estimate of the "true" value after the fact taking other measurements into account.
- knzhou 7y ago
  No, because the Bayesian conditioning ends up being used twice. If you and a friend do an experiment independently, and he gets 10 and you later get 12, you might justifiably think the answer is closer to 11. But you shouldn’t publish that you got 11. Because then somebody else will see that the measurements were 10 and 11, and think the true answer is closer to 10.5...
- o09rdk 7y ago
  I was kind of wondering about this after reading the passage. On the one hand the traditional Bayesian response is something like "yes, we're making our prior assumptions explicit and then incorporating that into a formal inferential paradigm." However, this prior is being used to bias the estimates, rather than to avoid the bias. That is, it would be akin to Dunnington taking the current estimates of e/m and using that to shape any new estimates from data. The argument is then that at least he was being explicit about his biases and how they are used to make an estimate. This has always seemed backwards to me, though. It seems what is more defensible is to use a formal theory about how prior biases affect estimates, and then to leverage that theory to minimize biases. This is basically the idea of the reference prior, to estimate things such that any role of the prior is minimized in an information-theoretic sense. This seems more analogous to what Dunnington was doing. I really wish reference priors were more widespread, although they can be computationally pretty hefty. It's one of my hopes that quantum computing might make these types of approaches more feasible in general.
SkyBelow 7y ago
> We’ve learned those tricks nowadays, and now we don’t have that kind of a disease. This isn't the case with sufficiently controversial topics in the social sciences.
- bsanr2 7y ago
  That would be because sufficiently controversial conclusions in the social sciences can upend social stability (endangering the ability of social science work to be performed) and serve as the justification for the infringement of human rights. That's bad, if there was any confusion. As another comment puts it, findings that buck conventional thought to the point that they can prompt irrevocable ramifications better have an "unimpeachable, reproducible path." And that's difficult when so much of social science is based on statistical modeling of phenomena that are multifaceted and difficult to control for, and not actually produced by deduction.
  - SkyBelow 7y ago
    >That would be because sufficiently controversial conclusions in the social sciences can upend social stability (endangering the ability of social science work to be performed) and serve as the justification for the infringement of human rights. This is assuming the current incorrect findings aren't being used as justifications for infringement of human rights currently. For example findings that might relate to rehabilitation for crimes and how long we sentence someone to prison, or if we even sentence someone to prison at all. Findings related to forced institutionalization of someone deemed too mentally ill to be allowed their freedom. Findings that are used to justify existing laws that see people put in prison for things they perhaps shouldn't be. Using such an impeccable standard for changes when such wasn't used for the existing social structure isn't justifiable.
    - shkkmo 7y ago
      > For example findings that might relate to rehabilitation for crimes and how long we sentence someone to prison, or if we even sentence someone to prison at all. Findings related to forced institutionalization of someone deemed too mentally ill to be allowed their freedom. Before cargo cult science becomes a problem here, we need to actually accept and adopt evidence based management of these institutions. Currently policies in these areas are primarily driven by tradition, rhetoric and anecdotes, not scientific inquiry.
deleted 7y ago
[deleted]
thyrsus 7y ago
I remember doing the Millikan oil drop experiment in a college physics lab. I got a charge value about 1/3 of what it should have been. Obviously, I was observing unpaired quarks ;'}
- plank 7y ago
  Interesting, I did so as well, back in - I think - 1990 or so. Did it together with a fellow student, we both reported seeing a quark, but did not get the Nobel prize;-)
- ahartmetz 7y ago
  I did it in a lab exercise and I think we were 20% off with the proper result somewhere close to the end of the error bar (inside or outside I don't remember). It felt disappointing at the time, like all early lab exercises. Some later ones were much closer to the state of the art.
gumby 7y ago
This kind of conservatism is actually pretty important. This is t engineering where you need to get that bridge built so people can use it; you’re trying to get to some “objective” “truth”. If your estimate is way off what someone else’s there’s a pretty good chance you’re wrong. After all the stepwise refinement did converge. The alternative leads to things like the cold fusion debacle. There are still people working on it, but the ones I know know that they are regarded as cranks (At least for their CF work) and are super careful to try and discount any results seemingly contradictory to existing physics. Which is correct: another CF claim better have an unimpeachable, reproducible path.
deleted 7y ago
[deleted]
Herrin 7y ago
I remember reading this a decade or two ago, but I never could find such a plot to confirm this. Fortunately, someone generated one here back in 2016: https://hsm.stackexchange.com/questions/264/timeline-of-measurements-of-the-electrons-charge https://hsm.stackexchange.com/questions/264/timeline-of-meas...

afthonos 7y ago

> We’ve learned those tricks nowadays, and now we don’t have that kind of a disease. That’s very optimistic. If nowhere else, polling has a huge herding problem, where outliers are dropped when they are too far off the consensus.

zamalek 7y ago
I'm pretty certain that's satire, or would be satire if Feynman were around to see the state of modern science (interesting things off-the-beaten-path were being done in his day).

chrisco255 7y ago

What evidence do you have that this sort of thing is not occurring nowadays?

leoc 7y ago
Well, the GP is quoting Feynman, so I'm not sure he's endorsing that part of the quotation. FWIW I think it's another healthy warning sign suggesting not to receive Feynman's pronouncements about scientific method too uncritically. However it may also be that the situation is now actually worse than when Feynman wrote this.

spats1990 7y ago

>One example of the principle is this: If you’ve made up your mind to test a theory, or you want to explain some idea, you should always decide to publish it whichever way it comes out. If we only publish results of a certain kind, we can make the argument look good. We must publish both kinds of result. Isn't non-publication of studies/research (for various reasons) a massive problem in a lot of different disciplines now? [1], for example. [1]https://pediatrics.aappublications.org/content/138/3/e20160223 https://pediatrics.aappublications.org/content/138/3/e201602...

bonoboTP 7y ago
Mainly because reviewers won't accept uninteresting results. In many fields it's hard to differentiate a negative result from just a clumsy researcher. In computer science a negative result usually generates a flood of questions like "okay, but why would you do it like that, couldn't you try this or that etc." and the conclusion is that the researcher just didn't manage to get it to work because they didn't use the right techniques. Essentially you'd have to argue that the method you tried should work, based on what we already know, but surprisingly it doesn't work. This type of argument is very hard to carry through. The opposite is much more publication friendly: "nobody would have thought that this could ever work like this, but we now show that this novel idea does work very well". That's the type of thing for award-winning research.
- lallysingh 7y ago
  Publication -- somewhere -- of interesting negative results would be invaluable specifically for the discussion of how the experiment went wrong. Experiment design is tricky and it's not something you get a lot of exposure to except through the experience of others you interact with.
  - sifar 7y ago
    Eliezer Yudkowsky talks about this in Inadequate Equiibria [1] when he analyzes academic research. short answer - here you need researchers and grant makers to simultaneously decide to do what is not in the best interest for them. [1] https://equilibriabook.com/an-equilibrium-of-no-free-energy/ https://equilibriabook.com/an-equilibrium-of-no-free-energy/

foldr 7y ago

>We’ve learned those tricks nowadays, and now we don’t have that kind of a disease. I'm sure the people doing those experiments also shared the conviction that they were smarter and more objective than people in past ages.

jstummbillig 7y ago
But we are getting better, right? Despite undoubtedly repeating a lot mistakes (and often times being foolishly ignorant of that), some collective knowledge is surely being retained, otherwise we would not make progress in basically all sciences.
- foldr 7y ago
  The fact that we're making progress in acquiring knowledge doesn't necessarily entail that we're getting better at being objective or unbiased. I see no reason to think that we are. (Nor am I particularly convinced that objectivity and lack of bias are major factors driving scientific progress, but that's probably another conversation.)
  - SiempreViernes 7y ago
    The very experiences that Feynman recounts have lead to a practice of doing "blind" experiments in particle physics and other places. You simply do your analysis with some extra, unknown, factor added and once you think you done the best you can, the blinding is removed and you check to see what you measured.
    - foldr 7y ago
      Sure, but who knows if this has been beneficial to scientific progress or not? It's not like we can wind back time and do a comparative experiment. edit To expand on this a bit, as it might sound flippant. People often talk about "bias" as if it's something that obviously ought to be eliminated. But in fact you can't get anywhere without biases. We only have the time and resources to explore a tiny portion of the hypothesis space in most domains.

dr_dshiv 7y ago

Isn't that also an appropriately Bayesian approach to new evidence?

marcosdumay 7y ago
No, it isn't. The appropriate approach is to accept the evidence, and correct your priors based on it, even if it's not good enough to believe. It's not to fiddle with the evidence until it's something you believe is correct. Of course, it's much easier said than done. I don't think any group of scientists is safe from repeating this.
SatvikBeri 7y ago
The correct approach would be to say something like "Previous experiments reported X, we observed X + Y, therefore the real answer is probably around X + Y/10." But it's very important to state the observation of X + Y, not X + Y/10, otherwise you'll take much too long to update to the correct value!
darkmighty 7y ago
Being a good Bayesian would entail taking the evidence as-is, only adjusting your estimate of the "true" value after the fact taking other measurements into account.
knzhou 7y ago
No, because the Bayesian conditioning ends up being used twice. If you and a friend do an experiment independently, and he gets 10 and you later get 12, you might justifiably think the answer is closer to 11. But you shouldn’t publish that you got 11. Because then somebody else will see that the measurements were 10 and 11, and think the true answer is closer to 10.5...
o09rdk 7y ago
I was kind of wondering about this after reading the passage. On the one hand the traditional Bayesian response is something like "yes, we're making our prior assumptions explicit and then incorporating that into a formal inferential paradigm." However, this prior is being used to bias the estimates, rather than to avoid the bias. That is, it would be akin to Dunnington taking the current estimates of e/m and using that to shape any new estimates from data. The argument is then that at least he was being explicit about his biases and how they are used to make an estimate. This has always seemed backwards to me, though. It seems what is more defensible is to use a formal theory about how prior biases affect estimates, and then to leverage that theory to minimize biases. This is basically the idea of the reference prior, to estimate things such that any role of the prior is minimized in an information-theoretic sense. This seems more analogous to what Dunnington was doing. I really wish reference priors were more widespread, although they can be computationally pretty hefty. It's one of my hopes that quantum computing might make these types of approaches more feasible in general.

SkyBelow 7y ago

> We’ve learned those tricks nowadays, and now we don’t have that kind of a disease. This isn't the case with sufficiently controversial topics in the social sciences.

bsanr2 7y ago
That would be because sufficiently controversial conclusions in the social sciences can upend social stability (endangering the ability of social science work to be performed) and serve as the justification for the infringement of human rights. That's bad, if there was any confusion. As another comment puts it, findings that buck conventional thought to the point that they can prompt irrevocable ramifications better have an "unimpeachable, reproducible path." And that's difficult when so much of social science is based on statistical modeling of phenomena that are multifaceted and difficult to control for, and not actually produced by deduction.
- SkyBelow 7y ago
  >That would be because sufficiently controversial conclusions in the social sciences can upend social stability (endangering the ability of social science work to be performed) and serve as the justification for the infringement of human rights. This is assuming the current incorrect findings aren't being used as justifications for infringement of human rights currently. For example findings that might relate to rehabilitation for crimes and how long we sentence someone to prison, or if we even sentence someone to prison at all. Findings related to forced institutionalization of someone deemed too mentally ill to be allowed their freedom. Findings that are used to justify existing laws that see people put in prison for things they perhaps shouldn't be. Using such an impeccable standard for changes when such wasn't used for the existing social structure isn't justifiable.
  - shkkmo 7y ago
    > For example findings that might relate to rehabilitation for crimes and how long we sentence someone to prison, or if we even sentence someone to prison at all. Findings related to forced institutionalization of someone deemed too mentally ill to be allowed their freedom. Before cargo cult science becomes a problem here, we need to actually accept and adopt evidence based management of these institutions. Currently policies in these areas are primarily driven by tradition, rhetoric and anecdotes, not scientific inquiry.

deleted 7y ago

[deleted]

thyrsus 7y ago

I remember doing the Millikan oil drop experiment in a college physics lab. I got a charge value about 1/3 of what it should have been. Obviously, I was observing unpaired quarks ;'}

plank 7y ago
Interesting, I did so as well, back in - I think - 1990 or so. Did it together with a fellow student, we both reported seeing a quark, but did not get the Nobel prize;-)
ahartmetz 7y ago
I did it in a lab exercise and I think we were 20% off with the proper result somewhere close to the end of the error bar (inside or outside I don't remember). It felt disappointing at the time, like all early lab exercises. Some later ones were much closer to the state of the art.

gumby 7y ago

This kind of conservatism is actually pretty important. This is t engineering where you need to get that bridge built so people can use it; you’re trying to get to some “objective” “truth”. If your estimate is way off what someone else’s there’s a pretty good chance you’re wrong. After all the stepwise refinement did converge. The alternative leads to things like the cold fusion debacle. There are still people working on it, but the ones I know know that they are regarded as cranks (At least for their CF work) and are super careful to try and discount any results seemingly contradictory to existing physics. Which is correct: another CF claim better have an unimpeachable, reproducible path.

deleted 7y ago

[deleted]

Herrin 7y ago

I remember reading this a decade or two ago, but I never could find such a plot to confirm this. Fortunately, someone generated one here back in 2016: https://hsm.stackexchange.com/questions/264/timeline-of-measurements-of-the-electrons-charge https://hsm.stackexchange.com/questions/264/timeline-of-meas...