9 ms·

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's

by misnome 10y ago

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs boson. The confirmation between ATLAS and CMS could be interpreted as merely internal cross-referencing - it is still using the same acceleration source. But everyone believes the results, and believes that they represent the Higgs. This isn't observed once in the experiment, it is observed many, many times, and very large amounts of scientists time are spent imagining, looking for, and measuring, any possible effect that could cause a distortion or bias to the data. When it costs billions to construct your experiment, sometimes reproducing the exact same thing can be hard. The same lengths are gone to in order to find alternate explanations or interpretations of the result data. If they don't, they know that some very hard questions are going to be asked - and there will be hard questions asked anyway, especially for extraordinary claims - look at e.g. DAMA/LIBRA which for years has observed what looks like indirect evidence for dark matter, but very few people actually believe it - the results remain unexplained whilst other experiments probe the same regions in different ways. Repetition is good, of course, but isn't a replacement for good science in the first place.

welterde 10y ago
ATLAS, CMS, ALICE, LHCb, etc. are all different experiments that just happen to share the same accelerator. They are done using different detectors, designed, built, operated and the data is analyzed by different groups of people. If you follow your logic, that the different experiments using the same accelerator negates the whole thing, to the extreme then doing two experiments on the same planet/solar system/universe won't be enough.. And I don't buy your inverse argument that one (good) experiment is good enough either. It is very difficult to tell from outside the group if the experiment is actually good or not (although you probably can tell if it's bad). Screw-ups can happen no matter how many people look at the data if there is some flaw in the experimental setup - only way to make really sure is to use different experiments to measure the same thing.
- misnome 10y ago
  I think you've got my argument completely backwards - I'm pointing out that if repetition as implied by the article is everything, then results from e.g. the LHC are discounted. Similarly, I am not saying that you don't need to repeat - just that it isn't the be all and end all of what defines 'science'. Supporting this interpretation is that I mentioned DAMA. Nobody is accusing them of not taking care, but nobody really believes the result either.
  - batbomb 10y ago
    In fact, CMS and ATLAS are sort of same experiment repeated twice, just in parallel rather than serially. Repeatable doesn't imply identical experimental design.
    - bordercases 10y ago
      Does it not? If the experimental design is not preserved, don't you open yourself to inconsistent controls between the experiments? Where do you draw the line?
      - shkkmo 10y ago
        If you think about it, complete replication of any experiment is useless, and impossible. But that doesn't matter, because it is not the experiment you are trying to replicate, is the effect or the observation. Carefully changing the design of the experiment can allow to verify that your explanation of the effect or observation is accurate. Of course, if the change the experiment too much, a failure to replicated will have less useful information.
        bordercases 10y ago
        In hindsight I do think it's true that you want numerous experiments with possibly different controls with the exception of whatever is important for the effect. In that sense you want your experiment to "discover the right abstraction" of details that your principle works over. So I think what has to occur is a gradual "loosening up" of the controls from strict replication to weak replication, with both types of replication giving information about the effect you are testing: its validity and its generality, respectively.
return0 10y ago
We don't need to 'define repetition', we need to foster a culture that a) accepts repetition and b) does not accept something for a fact just because it's in a journal. Right now, (a) is not even acceptable; nobody will publish a replication study. Ofc, LHC is impossible to replicate, but the vast majority of life science studies are. I should think this is mostly needed in life sciences. Other, more 'exact' sciences seem to not have this problem.
- throwaway729 10y ago
  > we need to foster a culture that a) accepts repetition Do we? I don't think we do. I think we need to foster a culture of honesty and rigor. Of good science. Which is decidedly different from fostering a culture of "repetition" for its own sake. Paying for the cost of mountains upon mountains of lab techs and materials that it would require to replicate every study published in a major journal just isn't a good use of ever-dwindling science dollars. Replicate where it's not far off the critical path. Replicate where the study is going to have a profound effect on the direction of research in several labs. But don't just replicate because "science!" In fact, one could argue that the increased strain on funding sources introduced by the huge cost of reproducing a bunch of stuff would increase the cut-throat culture of science and thereby decrease the scientist's natural proclivity toward honesty. > and b) does not accept something for a fact just because it's in a journal Again, it's entirely unclear what you mean here. It's impossible to re-verify every single paper you read (I've read three since breakfast). That would be like re-writing every single line of code of every dependency you pull into a project. And I'm pretty sure literally no scientist takes a paper's own description of its results at face value without reading through methods and looking at (at least) a summary of the data. Taking papers at face value is really only a problem in science reporting and at (very) sub-par institutions/venues. I don't care about the latter, and neither should you. WRT the former, science reporters often grossly misunderstand the paper anyways. All the good reproducible science in the world is of zero help if science reporters are going to bastardize the results beyond recognition anyways...
  - flubert 10y ago
    >ever-dwindling science dollars Do we have a good source of information on that? https://en.wikipedia.org/wiki/File:U.S._research_funding.png https://en.wikipedia.org/wiki/File:U.S._research_funding.png http://www.aaas.org/sites/default/files/DefNon%3B.jpg http://www.aaas.org/sites/default/files/DefNon%3B.jpg http://www.aaas.org/sites/default/files/Agencies%3B.jpg http://www.aaas.org/sites/default/files/Agencies%3B.jpg
johncolanduoni 10y ago
I think the difference is that we are able to test and understand/verify the operation of the LHC at multiple levels. It's not like they just built what was on paper, flicked the ON switch, and assumed all the data that came out was correct. The components were individually tested and collisions at energies accessible by other supercolliders were done so we could compare to other results. Unfortunately for medical drugs and psychology, researchers are mostly gathering data without an understanding of the underlying mechanisms. There are also virtually never have proposals which can be tested for compliance with reality in a quantifiable and isolated way as we can with physics, chemistry, or parts of biology. So I feel the replication crisis is not a matter of various fields just not knowing how to do "good science", but that these fields by their nature make clean hypothesis testing vastly more difficult and p-hacking and statistical trickery (intentional or otherwise) harder to sift out.
- return0 10y ago
  > but that these fields by their nature make clean hypothesis testing vastly more difficult and p-hacking and statistical trickery (intentional or otherwise) harder to sift out. How certain are we that its 'the nature' of these fields rather than political choices that were solidified long ago? There could be much larger samples and six-sigma confidence in life sciences if the grants were allocated differently. I know this is happening, for example, in neuroscience where the (private) Allen Institute is creating significantly more rigorous (and useful) datasets compared to the bulk of studies, because they are funding their studies differently.
zouhair 10y ago
You ask that like it is something new. Reproducibility[0] is something at the core of science. The problem these days is that spending time doing replication is not glamourous and will not help you get funds. [0]: https://en.wikipedia.org/wiki/Reproducibility https://en.wikipedia.org/wiki/Reproducibility
- analog31 10y ago
  I suppose the funding agencies could play a role in this. When you submit a proposal that cites prior work as a basis or motivation for you project, you should be required to show that either 1) The cited works have been reproduced, or 2) you're going to reproduce them.
  - zouhair 10y ago
    And also we need to start publishing negative studies way more.
    - analog31 10y ago
      I dimly recall some physics papers from years ago, with titles such as: "New experimental limit on the existence of..." Whenever my project was going particularly badly, we'd lighten things up by describing it as a negative study. One such highlight was: "New experimental limit on the existence of oil in the vacuum pump."
- Obi_Juan_Kenobi 10y ago
  The problem is worst in fields that don't have reproducibility 'built in' to the field. I do genetics and development, and the main sanity check we have is the distribution of mutant lines. If you say that mutant X does Y, other people are likely going to see that (or not) when they get their hands on it and start poking around. This strength of working with mutants is at the core of the success of molecular biology. Even if don't set out to confirm someone else's results, you're quite likely to come across some inconsistencies in the course of investigation. If a field lacks that sort of mechanism, they need to take special care to address reproducibility.
analog31 10y ago
Thinking about physics, the LHC experiment benefits from not being a completely isolated study. Instead, its results are multiple pieces of a larger jigsaw puzzle or web of interconnected results that include both a quantitative predictive theory and other experiments that test other predictions of the same theory. And the Higgs was not the only thing that could be tested at the LHC. For instance, it probably replicated a huge slew of prior studies during construction and testing of the system, from already-known particles all the way down to the calibration of voltmeters. In physics, some theories have been tested and interconnected to such a degree, that if an experiment conflicts with theory, it's reasonable to suspect the experiment rather than discard the theory. That's what happened with that apparent faster-than-light travel of neutrinos captured in Italy. It was something like a partially unplugged cable. Once corrections were made for the cable, the theory snapped right back into place. Those theories can be trusted as akin to tools in day to day physics. For my humble graduate experiment, refutation of any major law would have led me to fix the experiment and try again. I simply assumed things like conservation of charge. Such laws probably include electrodynamics, gravitation, quantum mechanics, thermodynamics, and Darwinian evolution. And this is a common feature of major studies in physics, chemistry, evolutionary biology, and other sciences as well. Where we may run into trouble is in branches of sciences that don't have over-arching theories or that web of connected results. At the other end of the scale are areas of sciences where the results are mainly a database of observed statistical correlations, with little or no apparent progress towards a general theory. When someone publishes a surprising result, there is no reason to say that the experiment must have been done wrong. You just add it to the pile. The best that can be hoped for is that some kind of meta-analysis will demonstrate an inconsistency among multiple studies. Those fields don't have the "hard" theories that can be used as tools to test experimental results. To be fair, there's another situation, where you have not one, but zero, reproducible results. That's the state of affairs in the search to unify gravity and quantum mechanics.
dschiptsov 10y ago
What has been seen is reading of instruments which has been built according to some mathematical model to register conditions which supposed to happen according to bunch of theories and then a lot of calculations follows which presumably conform the assumptions according to the reading. The simple analogy is pictures one has on fMRI. These pictures are, well, pictures based on approximations, not the mind. Not even close. There is a paradox - one cannot make an instrument out of atoms to "see" what is going on inside these atoms. What they saw are pictures created out of mathematical models, not reality as it is.
danielmorozoff 10y ago
With all respect I do not agree with your example for the lhc . For two reasons primarily: The first has already been mentioned about various groups and experiments / detectors. The second, I think is more important. In order to claim discoveries at any of those experiments certain statistical constraints must be met. The 5 sigma rule is by its very nature garnered by repetition and is a necessary because of the nature of quantum mechanics and particle physics. Here is a write up on cern and 5 sigma http://physicsbuzz.physicscentral.com/2012/07/does-5-sigma-discovery.html?m=1 http://physicsbuzz.physicscentral.com/2012/07/does-5-sigma-d...
Zigurd 10y ago
Not that it isn't important in physics, it is important but for different reasons. Nobody is coming around selling physicists anything based on a non-repeated experiment. Drug studies has a lot of perverse incentives in play, and the subsequent interpretation of results for, for example, off-label use are also skewed toward one set of interpretations. There could be billions of dollars in sales at stake.
peter303 10y ago
There were several "bogey Higgs" at different energies during earlier Tevatron and LHC runs. Maybe these were 3-5 events which hinted at a signal. Then they disappeared. But the real Higgs was not confirmed until there were 40 events of two different decay paths at two different detectors, i.e. 5-sigma significance. Different runs and decay paths are like multiple replications.
anotherhacker 10y ago
The difference is maths. E.g. Einstein proved his theory of gravity mathematically, THEN it was proved via empirical observation. We don't have a way to mathematically prove a drug for depression a psychological phenomena.
- johncolanduoni 10y ago
  What does "prove mathematically" mean here? He constructed a self-consistent mathematical theory, and validated that it reproduced both the special relativistic and Newtonian limits, but I don't think "proved mathematically" is a good way to state that. You can't "mathematically prove" a theory agrees with reality except by reconciling it with experiments (possibly indirectly through links to previous theories), so the only difference here is that a non-ad-hoc mathematical theory of depression medication doesn't appear to be feasible, not that physics has some magic alternate method of proof.
  - anotherhacker 10y ago
    Did you google it? https://en.wikipedia.org/wiki/Mathematical_proof https://en.wikipedia.org/wiki/Mathematical_proof You're missing the point of what I said w/ the Einstein example. He proved it via maths. Whether or not it was empirically validated is irrelevant. It was true mathematically Today, this cannot be done with fields like medicine and psychology. No one can create a mathematical proof for a cure for cancer, make a treatment based upon that proof, and have it work on the first try
    - johncolanduoni 10y ago
      I've done enough mathematical proofs that I don't think I need to "google it", but I read the Wikipedia page anyway. It failed to explain how a mathematical proof could have anything to do with "proving" General Relativity in absence of empirical evidence. He certainly proved that (as I said before) it agreed with existing (experimentally backed) theories using mathematics, but if you have a method which allows you to deduce the truth of physical laws from purely mathematical proofs (no links to experiments at all!) you'll revolutionize epistemology. As it stands most scientists are using the crutch of experimental evidence at some level (either directly or by reasoning within theories that are at least for now validated by experiment). > Whether or not it was empirically validated is irrelevant. It was true mathematically If you mean it was a self-consistent theory, note that there are infinite physically incorrect theories which are "true mathematically". It is true mathematically in the same way number theory is "true mathematically", but you don't see anyone assume that we can describe gravity with prime number theory. If you mean it was mathematically proved consistent with previous physical theories backed by evidence, we're back to the experimental link, and also there are infinite physically incorrect theories which would agree with Newtonian mechanics and special relativity. If you mean he proved it satisfied some properties we expect from physical theories because they've been consistently upheld (e.g. energy conservation), there's an infinite number of wrong ones there too. Mathematical convenience has been an excellent guide in physics, particularly fundamental physics (electromagnetic waves, antiparticles, and the Higgs Boson were all the results of positing things for mathematical convenience), but it is not a substitute for verifying novel experimental predictions. Nobody taken seriously in the physics (or mathematics for that matter!) community thinks it is, not even the often decried string theorists.
macspoofing 10y ago
>The confirmation between ATLAS and CMS could be interpreted as merely internal cross-referencing Why? ATLAS and CMS are different detectors operated by different teams.
unfamiliar 10y ago
The LHC is somewhat of a special case, because repetition is extremely difficult for obvious reasons. That doesn't preclude making repetition an appropriate requirement in 99.9% of other scientific findings. In this particular case, we just have to make sure that the LHC guys conform to extremely stringent burdens of proof and high scientific accountability, which they already impose on themselves but which the 99.9% do not impose on themselves for a variety of reasons (an uncomfortable truth, but due to many things like pressure to publish and generally being less good, a truth nonetheless).
yk 10y ago
Physics is a special case for several reasons, [1] most importantly there is a good theory with the effect that different experiments measure the same thing. So once you start looking for a specific deviation from the standard model, you can not only look at accelerator experiments, but also at g-2 experiments [2], or lamp shift or beta decay in the case of the electro-weak interaction. (Or at different places in accelerator experiments, in the sense that data at one point should lead to corrections at another point.) In other disciplines, the role of repetition is a lot more important. The look elsewhere effect means, that your statistical significance depends on all other research, published or unpublished. [3] If you are doing a repetition study, the look elsewhere effect just goes away. So good science in the first place is very important of course, but there is a lot of value in doing repeated experiments for fundamental statistical reasons. Especially in fields were there is no good predictive theory (everywhere except physics) and where it is very hard to go to very high significance. [1] I recently read John Lewis Gaddis, The Landscape of History, 2004 which is a essay on epistemology of history and in a way tries to reject "physics envy," with the argument that physics is everything where there is a good theory, so in a way where we are winning. And consequently physics envy looks a lot like cargo cult, other disciplines need to figure out their own epistemology. [2] Incredible high precision experiments which measure the magnetic moment of electrons. [3] https://xkcd.com/882/ https://xkcd.com/882/
jsprogrammer 10y ago
>Repetition is good, of course, but isn't a replacement for good science in the first place. Repetition is science. >Define repetition. Google says: "the recurrence of an action or event." Science requires 1) a prediction of an observation 2) repeated observation consistent with the prediction. If there is no repeated observation of what was predicted, it is not science. >But everyone believes the results, and believes that they represent the Higgs. It doesn't matter what is believed. "Everyone" used to believe all kinds of things, that doesn't mean that those things were true, or accurate. Repeated observation of prior predictions is all that matters. >When it costs billions to construct your experiment, sometimes reproducing the exact same thing can be hard. That doesn't excuse, or allow in, things that haven't been reproduced.
boznz 10y ago
Repetition is given all the facts and original conditions the same or similar result can be expected. Physics has its own cases such as cold fusion which is unreproducable and should not be if all the above hold true.
StanislavPetrov 10y ago
>It's not as simple as that, for all sciences Yes, it is - that's the very basis for science. One of the main problems is that many academic disciplines have been wrongly classified as hard science (see "social" sciences). >The LHC has only been built once.. But everyone believes the results, and believes that they represent the Higgs. Nobody intelligent "believes" anything. We examine evidence and draw tentative conclusions based on that evidence, always retaining doubt because, unless you are omniscient, there is always new information that can come to light that can cause you to change your conclusion. Science is a process. If you "believe" anything without doubt, you aren't a scientist, you are a priest.

by misnome 10y ago

welterde 10y ago

ATLAS, CMS, ALICE, LHCb, etc. are all different experiments that just happen to share the same accelerator. They are done using different detectors, designed, built, operated and the data is analyzed by different groups of people. If you follow your logic, that the different experiments using the same accelerator negates the whole thing, to the extreme then doing two experiments on the same planet/solar system/universe won't be enough.. And I don't buy your inverse argument that one (good) experiment is good enough either. It is very difficult to tell from outside the group if the experiment is actually good or not (although you probably can tell if it's bad). Screw-ups can happen no matter how many people look at the data if there is some flaw in the experimental setup - only way to make really sure is to use different experiments to measure the same thing.

misnome 10y ago
I think you've got my argument completely backwards - I'm pointing out that if repetition as implied by the article is everything, then results from e.g. the LHC are discounted. Similarly, I am not saying that you don't need to repeat - just that it isn't the be all and end all of what defines 'science'. Supporting this interpretation is that I mentioned DAMA. Nobody is accusing them of not taking care, but nobody really believes the result either.
- batbomb 10y ago
  In fact, CMS and ATLAS are sort of same experiment repeated twice, just in parallel rather than serially. Repeatable doesn't imply identical experimental design.
  - bordercases 10y ago
    Does it not? If the experimental design is not preserved, don't you open yourself to inconsistent controls between the experiments? Where do you draw the line?
    - shkkmo 10y ago
      If you think about it, complete replication of any experiment is useless, and impossible. But that doesn't matter, because it is not the experiment you are trying to replicate, is the effect or the observation. Carefully changing the design of the experiment can allow to verify that your explanation of the effect or observation is accurate. Of course, if the change the experiment too much, a failure to replicated will have less useful information.
      - bordercases 10y ago
        In hindsight I do think it's true that you want numerous experiments with possibly different controls with the exception of whatever is important for the effect. In that sense you want your experiment to "discover the right abstraction" of details that your principle works over. So I think what has to occur is a gradual "loosening up" of the controls from strict replication to weak replication, with both types of replication giving information about the effect you are testing: its validity and its generality, respectively.

return0 10y ago

We don't need to 'define repetition', we need to foster a culture that a) accepts repetition and b) does not accept something for a fact just because it's in a journal. Right now, (a) is not even acceptable; nobody will publish a replication study. Ofc, LHC is impossible to replicate, but the vast majority of life science studies are. I should think this is mostly needed in life sciences. Other, more 'exact' sciences seem to not have this problem.

throwaway729 10y ago
> we need to foster a culture that a) accepts repetition Do we? I don't think we do. I think we need to foster a culture of honesty and rigor. Of good science. Which is decidedly different from fostering a culture of "repetition" for its own sake. Paying for the cost of mountains upon mountains of lab techs and materials that it would require to replicate every study published in a major journal just isn't a good use of ever-dwindling science dollars. Replicate where it's not far off the critical path. Replicate where the study is going to have a profound effect on the direction of research in several labs. But don't just replicate because "science!" In fact, one could argue that the increased strain on funding sources introduced by the huge cost of reproducing a bunch of stuff would increase the cut-throat culture of science and thereby decrease the scientist's natural proclivity toward honesty. > and b) does not accept something for a fact just because it's in a journal Again, it's entirely unclear what you mean here. It's impossible to re-verify every single paper you read (I've read three since breakfast). That would be like re-writing every single line of code of every dependency you pull into a project. And I'm pretty sure literally no scientist takes a paper's own description of its results at face value without reading through methods and looking at (at least) a summary of the data. Taking papers at face value is really only a problem in science reporting and at (very) sub-par institutions/venues. I don't care about the latter, and neither should you. WRT the former, science reporters often grossly misunderstand the paper anyways. All the good reproducible science in the world is of zero help if science reporters are going to bastardize the results beyond recognition anyways...
- flubert 10y ago
  >ever-dwindling science dollars Do we have a good source of information on that? https://en.wikipedia.org/wiki/File:U.S._research_funding.png https://en.wikipedia.org/wiki/File:U.S._research_funding.png http://www.aaas.org/sites/default/files/DefNon%3B.jpg http://www.aaas.org/sites/default/files/DefNon%3B.jpg http://www.aaas.org/sites/default/files/Agencies%3B.jpg http://www.aaas.org/sites/default/files/Agencies%3B.jpg

johncolanduoni 10y ago

I think the difference is that we are able to test and understand/verify the operation of the LHC at multiple levels. It's not like they just built what was on paper, flicked the ON switch, and assumed all the data that came out was correct. The components were individually tested and collisions at energies accessible by other supercolliders were done so we could compare to other results. Unfortunately for medical drugs and psychology, researchers are mostly gathering data without an understanding of the underlying mechanisms. There are also virtually never have proposals which can be tested for compliance with reality in a quantifiable and isolated way as we can with physics, chemistry, or parts of biology. So I feel the replication crisis is not a matter of various fields just not knowing how to do "good science", but that these fields by their nature make clean hypothesis testing vastly more difficult and p-hacking and statistical trickery (intentional or otherwise) harder to sift out.

return0 10y ago
> but that these fields by their nature make clean hypothesis testing vastly more difficult and p-hacking and statistical trickery (intentional or otherwise) harder to sift out. How certain are we that its 'the nature' of these fields rather than political choices that were solidified long ago? There could be much larger samples and six-sigma confidence in life sciences if the grants were allocated differently. I know this is happening, for example, in neuroscience where the (private) Allen Institute is creating significantly more rigorous (and useful) datasets compared to the bulk of studies, because they are funding their studies differently.

zouhair 10y ago

You ask that like it is something new. Reproducibility[0] is something at the core of science. The problem these days is that spending time doing replication is not glamourous and will not help you get funds. [0]: https://en.wikipedia.org/wiki/Reproducibility https://en.wikipedia.org/wiki/Reproducibility

analog31 10y ago
I suppose the funding agencies could play a role in this. When you submit a proposal that cites prior work as a basis or motivation for you project, you should be required to show that either 1) The cited works have been reproduced, or 2) you're going to reproduce them.
- zouhair 10y ago
  And also we need to start publishing negative studies way more.
  - analog31 10y ago
    I dimly recall some physics papers from years ago, with titles such as: "New experimental limit on the existence of..." Whenever my project was going particularly badly, we'd lighten things up by describing it as a negative study. One such highlight was: "New experimental limit on the existence of oil in the vacuum pump."
Obi_Juan_Kenobi 10y ago
The problem is worst in fields that don't have reproducibility 'built in' to the field. I do genetics and development, and the main sanity check we have is the distribution of mutant lines. If you say that mutant X does Y, other people are likely going to see that (or not) when they get their hands on it and start poking around. This strength of working with mutants is at the core of the success of molecular biology. Even if don't set out to confirm someone else's results, you're quite likely to come across some inconsistencies in the course of investigation. If a field lacks that sort of mechanism, they need to take special care to address reproducibility.

analog31 10y ago

Thinking about physics, the LHC experiment benefits from not being a completely isolated study. Instead, its results are multiple pieces of a larger jigsaw puzzle or web of interconnected results that include both a quantitative predictive theory and other experiments that test other predictions of the same theory. And the Higgs was not the only thing that could be tested at the LHC. For instance, it probably replicated a huge slew of prior studies during construction and testing of the system, from already-known particles all the way down to the calibration of voltmeters. In physics, some theories have been tested and interconnected to such a degree, that if an experiment conflicts with theory, it's reasonable to suspect the experiment rather than discard the theory. That's what happened with that apparent faster-than-light travel of neutrinos captured in Italy. It was something like a partially unplugged cable. Once corrections were made for the cable, the theory snapped right back into place. Those theories can be trusted as akin to tools in day to day physics. For my humble graduate experiment, refutation of any major law would have led me to fix the experiment and try again. I simply assumed things like conservation of charge. Such laws probably include electrodynamics, gravitation, quantum mechanics, thermodynamics, and Darwinian evolution. And this is a common feature of major studies in physics, chemistry, evolutionary biology, and other sciences as well. Where we may run into trouble is in branches of sciences that don't have over-arching theories or that web of connected results. At the other end of the scale are areas of sciences where the results are mainly a database of observed statistical correlations, with little or no apparent progress towards a general theory. When someone publishes a surprising result, there is no reason to say that the experiment must have been done wrong. You just add it to the pile. The best that can be hoped for is that some kind of meta-analysis will demonstrate an inconsistency among multiple studies. Those fields don't have the "hard" theories that can be used as tools to test experimental results. To be fair, there's another situation, where you have not one, but zero, reproducible results. That's the state of affairs in the search to unify gravity and quantum mechanics.

dschiptsov 10y ago

What has been seen is reading of instruments which has been built according to some mathematical model to register conditions which supposed to happen according to bunch of theories and then a lot of calculations follows which presumably conform the assumptions according to the reading. The simple analogy is pictures one has on fMRI. These pictures are, well, pictures based on approximations, not the mind. Not even close. There is a paradox - one cannot make an instrument out of atoms to "see" what is going on inside these atoms. What they saw are pictures created out of mathematical models, not reality as it is.

danielmorozoff 10y ago

With all respect I do not agree with your example for the lhc . For two reasons primarily: The first has already been mentioned about various groups and experiments / detectors. The second, I think is more important. In order to claim discoveries at any of those experiments certain statistical constraints must be met. The 5 sigma rule is by its very nature garnered by repetition and is a necessary because of the nature of quantum mechanics and particle physics. Here is a write up on cern and 5 sigma http://physicsbuzz.physicscentral.com/2012/07/does-5-sigma-discovery.html?m=1 http://physicsbuzz.physicscentral.com/2012/07/does-5-sigma-d...

Zigurd 10y ago

Not that it isn't important in physics, it is important but for different reasons. Nobody is coming around selling physicists anything based on a non-repeated experiment. Drug studies has a lot of perverse incentives in play, and the subsequent interpretation of results for, for example, off-label use are also skewed toward one set of interpretations. There could be billions of dollars in sales at stake.

peter303 10y ago

There were several "bogey Higgs" at different energies during earlier Tevatron and LHC runs. Maybe these were 3-5 events which hinted at a signal. Then they disappeared. But the real Higgs was not confirmed until there were 40 events of two different decay paths at two different detectors, i.e. 5-sigma significance. Different runs and decay paths are like multiple replications.

anotherhacker 10y ago

The difference is maths. E.g. Einstein proved his theory of gravity mathematically, THEN it was proved via empirical observation. We don't have a way to mathematically prove a drug for depression a psychological phenomena.

johncolanduoni 10y ago
What does "prove mathematically" mean here? He constructed a self-consistent mathematical theory, and validated that it reproduced both the special relativistic and Newtonian limits, but I don't think "proved mathematically" is a good way to state that. You can't "mathematically prove" a theory agrees with reality except by reconciling it with experiments (possibly indirectly through links to previous theories), so the only difference here is that a non-ad-hoc mathematical theory of depression medication doesn't appear to be feasible, not that physics has some magic alternate method of proof.
- anotherhacker 10y ago
  Did you google it? https://en.wikipedia.org/wiki/Mathematical_proof https://en.wikipedia.org/wiki/Mathematical_proof You're missing the point of what I said w/ the Einstein example. He proved it via maths. Whether or not it was empirically validated is irrelevant. It was true mathematically Today, this cannot be done with fields like medicine and psychology. No one can create a mathematical proof for a cure for cancer, make a treatment based upon that proof, and have it work on the first try
  - johncolanduoni 10y ago
    I've done enough mathematical proofs that I don't think I need to "google it", but I read the Wikipedia page anyway. It failed to explain how a mathematical proof could have anything to do with "proving" General Relativity in absence of empirical evidence. He certainly proved that (as I said before) it agreed with existing (experimentally backed) theories using mathematics, but if you have a method which allows you to deduce the truth of physical laws from purely mathematical proofs (no links to experiments at all!) you'll revolutionize epistemology. As it stands most scientists are using the crutch of experimental evidence at some level (either directly or by reasoning within theories that are at least for now validated by experiment). > Whether or not it was empirically validated is irrelevant. It was true mathematically If you mean it was a self-consistent theory, note that there are infinite physically incorrect theories which are "true mathematically". It is true mathematically in the same way number theory is "true mathematically", but you don't see anyone assume that we can describe gravity with prime number theory. If you mean it was mathematically proved consistent with previous physical theories backed by evidence, we're back to the experimental link, and also there are infinite physically incorrect theories which would agree with Newtonian mechanics and special relativity. If you mean he proved it satisfied some properties we expect from physical theories because they've been consistently upheld (e.g. energy conservation), there's an infinite number of wrong ones there too. Mathematical convenience has been an excellent guide in physics, particularly fundamental physics (electromagnetic waves, antiparticles, and the Higgs Boson were all the results of positing things for mathematical convenience), but it is not a substitute for verifying novel experimental predictions. Nobody taken seriously in the physics (or mathematics for that matter!) community thinks it is, not even the often decried string theorists.

macspoofing 10y ago

>The confirmation between ATLAS and CMS could be interpreted as merely internal cross-referencing Why? ATLAS and CMS are different detectors operated by different teams.

unfamiliar 10y ago

The LHC is somewhat of a special case, because repetition is extremely difficult for obvious reasons. That doesn't preclude making repetition an appropriate requirement in 99.9% of other scientific findings. In this particular case, we just have to make sure that the LHC guys conform to extremely stringent burdens of proof and high scientific accountability, which they already impose on themselves but which the 99.9% do not impose on themselves for a variety of reasons (an uncomfortable truth, but due to many things like pressure to publish and generally being less good, a truth nonetheless).

yk 10y ago

Physics is a special case for several reasons, [1] most importantly there is a good theory with the effect that different experiments measure the same thing. So once you start looking for a specific deviation from the standard model, you can not only look at accelerator experiments, but also at g-2 experiments [2], or lamp shift or beta decay in the case of the electro-weak interaction. (Or at different places in accelerator experiments, in the sense that data at one point should lead to corrections at another point.) In other disciplines, the role of repetition is a lot more important. The look elsewhere effect means, that your statistical significance depends on all other research, published or unpublished. [3] If you are doing a repetition study, the look elsewhere effect just goes away. So good science in the first place is very important of course, but there is a lot of value in doing repeated experiments for fundamental statistical reasons. Especially in fields were there is no good predictive theory (everywhere except physics) and where it is very hard to go to very high significance. [1] I recently read John Lewis Gaddis, The Landscape of History, 2004 which is a essay on epistemology of history and in a way tries to reject "physics envy," with the argument that physics is everything where there is a good theory, so in a way where we are winning. And consequently physics envy looks a lot like cargo cult, other disciplines need to figure out their own epistemology. [2] Incredible high precision experiments which measure the magnetic moment of electrons. [3] https://xkcd.com/882/ https://xkcd.com/882/

jsprogrammer 10y ago

>Repetition is good, of course, but isn't a replacement for good science in the first place. Repetition is science. >Define repetition. Google says: "the recurrence of an action or event." Science requires 1) a prediction of an observation 2) repeated observation consistent with the prediction. If there is no repeated observation of what was predicted, it is not science. >But everyone believes the results, and believes that they represent the Higgs. It doesn't matter what is believed. "Everyone" used to believe all kinds of things, that doesn't mean that those things were true, or accurate. Repeated observation of prior predictions is all that matters. >When it costs billions to construct your experiment, sometimes reproducing the exact same thing can be hard. That doesn't excuse, or allow in, things that haven't been reproduced.

boznz 10y ago

Repetition is given all the facts and original conditions the same or similar result can be expected. Physics has its own cases such as cold fusion which is unreproducable and should not be if all the above hold true.

StanislavPetrov 10y ago

>It's not as simple as that, for all sciences Yes, it is - that's the very basis for science. One of the main problems is that many academic disciplines have been wrongly classified as hard science (see "social" sciences). >The LHC has only been built once.. But everyone believes the results, and believes that they represent the Higgs. Nobody intelligent "believes" anything. We examine evidence and draw tentative conclusions based on that evidence, always retaining doubt because, unless you are omniscient, there is always new information that can come to light that can cause you to change your conclusion. Science is a process. If you "believe" anything without doubt, you aren't a scientist, you are a priest.