3 ms·

Absence of evidence

56 points by henning 16y ago

_delirium 16y ago
If you're going to take it as an absolute statement, sure. But the more usual meaning in science, at least, is that the lack of evidence for something doesn't by itself constitute evidence against it (or at least, it's evidence with weight arbitrarily close to zero). You need something more, such as absence of evidence and a reason to believe that finding such evidence, if it existed, would have been at least somewhat probable. (Which is of course why the example in this post is doing the calculation of what observations we would've expected under each scenario.)
- pjscott 16y ago
  Sometimes I think this would be a lot easier for everyone if we just hauled out Bayes' theorem and had done with it. If a hypothesis (weighted with your confidence in it) predicts that a set of observations would turn up more evidence for it than you actually find, then that is evidence against that hypothesis, and should cause you to lower your estimate of the probability that the hypothesis is correct. And vice versa. For example, suppose that I want to explain why my apples have been stolen from my apartment. I live in a hypothetical universe where there are only three possibilities, all of which have roughly the same likelihood, a priori: spontaneous combustion, alien abduction, or theft by a troupe of dastardly harlequins. If the apples combusted, I would expect to see scorch marks, or perhaps soot stains on the wall nearby. If they were abducted by aliens, I would not expect to see anything, since everyone knows that alien abduction rays leave no signs. Finally, if the vile harlequins made off with my fruit, I would expect to see some signs of their passing -- face-paint on the doorknob, perhaps a fool's cap hastily discarded in a corner. Now, I inspect my apartment. I find no scorch marks, which is evidence against spontaneous combustion. If my apples had burst into flame, it's technically possible that they could have done so without leaving marks, but unlikely. I find no evidence of alien abduction, which is exactly what I would expect to find, so it does not change my estimate of that hypothesis' likelihood. Finally, I find a scrap of ripped, brightly-colored cloth which appears to have been torn from the outfit of an interloper hastily scurrying away through my pointlessly-large ventilation ducts. My face grows grim as I raise my estimate of the likelihood of my third hypothesis. Harlequins! Foul creatures of the Commedia dell'arte! My apartment has been defiled by their acrobatics and shenanigans! And they stole my darn apples! So, in the case of spontaneous combustion, absence of evidence was evidence of absence. For alien fruit abduction, absence of evidence wasn't evidence of anything. In the third case, there was evidence above what I would expect from my prior beliefs, so that was evidence for the harlequin hypothesis.
  - _delirium 16y ago
    I buy that, but I guess to me that is what the slogan is supposed to imply, but I do agree it's not always taken that way. That is: merely not finding evidence tells you nothing. However, not finding evidence, when you have a prediction of what evidence should be found, may be evidence against. So you need two things there: 1) absence of evidence; and 2) a prediction of what evidence there should've been. If all you have is #1 by itself, you have no evidence of anything. Hence, absence of evidence (without more) isn't evidence of absence. Maybe I'm in the wrong debates, but most of the times when I see the slogan applied, the person doesn't have the second part. For example, a common elementary error is to conclude that variables are uncorrelated because you calculated their correlation in your data set and it was not significant. Failure to find a correlation in a very large population study that's large enough that it should have found one, can be evidence of no correlation (or no large correlation, at least), but you have to do a separate analysis for that.
smanek 16y ago
It's a pretty straightforward implication of bayes' law that the absence of evidence is evidence of absence. In fact, we can quite easily quantify just how strong the evidence it provides is. http://lesswrong.com/lw/ih/absence_of_evidence_is_evidence_of_absence/ http://lesswrong.com/lw/ih/absence_of_evidence_is_evidence_o...
- PaulHoule 16y ago
  But, with Bayesian methods you're feeding in prior information which can itself be flawed. When it comes to some question like 'What is reliability of the space shuttle?' the largest uncertainty in the calculation is that there is some failure mode you haven't thought of. Despite decades of changes to the space shuttle to improve safety, the official estimate of the failure probability hasn't changed since 1975. In conventional logic, we say that a reasoning method is sound if it never comes to correct conclusions. One version of 'A of E' -> 'E of A' is the "closed world assumption" which, in one variant, means that "if you haven't been presented with X, then not(X) is true." Now, we know that's not sound, not correct 100% of the time. However, when doing 'commonsense reasoning', the CWA can sometimes be a highly effective heuristic. (Note also that's there's no completely satisfactory way to deal with logical negation in knoweldge based systems at this time, so we've got to choose some poison, or otherwise deal with a system that "can't say no". Part of that is that 'commonsense reasoning' is probabilistic. For instance, an effective hold 'em player has to estimate the probability of improving his hand at drawing, have a guess about what his opponent(s) may be holding, and how the opponent may respond to his actions. From that he's able to estimate odds which help him decide what to bet. Poker's a pure example, but the same kind of probabilistic reasoning is necessary for hunting, software project management, scientific research, and many of things that people do.
- amalcon 16y ago
  That article is subtly wrong. Certainly it is less wrong (pun intended) than the reasoning it criticizes, but it does subtly miss the mark. Bayes' Law refers to data: specifically, it relates data to evidence. When a datum provides evidence for a hypothesis, the reverse of that datum must also provide evidence against that hypothesis. That is to say, a datum is either relevant or it's not. If it is relevant, then it will present evidence either for or against a given hypothesis, depending on its value. If you don't know that datum, however, it manages to not provide evidence either way. As such, it is easy to lack evidence either for or against a hypothesis. Through straightforward logical manipulations from there, it's possible to have "absence of evidence" while not having "evidence of absence". Therefore, they cannot be equivalent.
baddox 16y ago
The big point here, which the author doesn't mention, is that you can have absence of evidence without searching for evidence or you can have absence of evidence after rigorous searching. I believe that the former situation is the one covered by the saying "absence of evidence is not evidence of absence." For example, I personally haven't observed any evidence of gravitational time dilation, because I haven't looked. I would be a fool to interpret this lack of evidence as evidence that gravitational time dilation doesn't exist. The engineers of the Global Positioning System have evidence. However, after a reasonably rigorous search, one may indeed reasonably conclude that something doesn't exist if no evidence of it has been found, especially when evidence would be easily visible if the thing did exist. The examples the author gave fall into this category: Most likely, given the distribution of humans on Earth, there would be evidence of dodo birds if they were extant. Now, it's important to be careful when estimating the likelihood of stumbling upon evidence of something. Hundreds of new species (most only subtly different than known species) are discovered every year, and there are undoubtedly thousands of oceanic creatures that have never been seen by humans, much less captured and dissected. It would be foolish to deny that there aren't hundreds or thousands of yet-undiscovered species simply because as of now we have no evidence of them.
tjmaxal 16y ago
The turkey comment reminds me of black swan investing. Just because it hasn't happened yet doesn't mean it won't in fact for certain things, it actually increases the odds that it will happen the more time passes without it occurring, like Volcanic eruptions.
Silhouette 16y ago
Another slightly annoying saying is "The plural of anecdote is not data". Any anecdote is a data point, if you assume that it is factually correct. You may or may not be able to draw any significant conclusions from that data, once you take into account sample size and bias, of course. But unless you're going to "sample" the entire population of interest, pretty much all statistical analysis in the real world is done based on multiple anecdotes. We just try to minimise the bias and to choose a sample size large enough to give results significant enough for our purposes.
- Panoramix 16y ago
  I disagree. I interpret that phrase as stating that an anecdote is something fundamentally different from a valid data point: the data point came from a carefully designed experiment using properly tested equipment, control groups, rigorous mathematical analysis, perhaps using double-blind, randomized trials where it applies, etc... whereas the anecdote is a story you hear from somebody, as per its definition: http://en.wikipedia.org/wiki/Anecdotal_evidence http://en.wikipedia.org/wiki/Anecdotal_evidence
- wglb 16y ago
  Ah, but you see, there are those that say that "data" by definition is plural.

56 points by henning 16y ago

If you're going to take it as an absolute statement, sure. But the more usual meaning in science, at least, is that the lack of evidence for something doesn't by itself constitute evidence against it (or at least, it's evidence with weight arbitrarily close to zero). You need something more, such as absence of evidence and a reason to believe that finding such evidence, if it existed, would have been at least somewhat probable. (Which is of course why the example in this post is doing the calculation of what observations we would've expected under each scenario.)

pjscott 16y ago
Sometimes I think this would be a lot easier for everyone if we just hauled out Bayes' theorem and had done with it. If a hypothesis (weighted with your confidence in it) predicts that a set of observations would turn up more evidence for it than you actually find, then that is evidence against that hypothesis, and should cause you to lower your estimate of the probability that the hypothesis is correct. And vice versa. For example, suppose that I want to explain why my apples have been stolen from my apartment. I live in a hypothetical universe where there are only three possibilities, all of which have roughly the same likelihood, a priori: spontaneous combustion, alien abduction, or theft by a troupe of dastardly harlequins. If the apples combusted, I would expect to see scorch marks, or perhaps soot stains on the wall nearby. If they were abducted by aliens, I would not expect to see anything, since everyone knows that alien abduction rays leave no signs. Finally, if the vile harlequins made off with my fruit, I would expect to see some signs of their passing -- face-paint on the doorknob, perhaps a fool's cap hastily discarded in a corner. Now, I inspect my apartment. I find no scorch marks, which is evidence against spontaneous combustion. If my apples had burst into flame, it's technically possible that they could have done so without leaving marks, but unlikely. I find no evidence of alien abduction, which is exactly what I would expect to find, so it does not change my estimate of that hypothesis' likelihood. Finally, I find a scrap of ripped, brightly-colored cloth which appears to have been torn from the outfit of an interloper hastily scurrying away through my pointlessly-large ventilation ducts. My face grows grim as I raise my estimate of the likelihood of my third hypothesis. Harlequins! Foul creatures of the Commedia dell'arte! My apartment has been defiled by their acrobatics and shenanigans! And they stole my darn apples! So, in the case of spontaneous combustion, absence of evidence was evidence of absence. For alien fruit abduction, absence of evidence wasn't evidence of anything. In the third case, there was evidence above what I would expect from my prior beliefs, so that was evidence for the harlequin hypothesis.
- _delirium 16y ago
  I buy that, but I guess to me that is what the slogan is supposed to imply, but I do agree it's not always taken that way. That is: merely not finding evidence tells you nothing. However, not finding evidence, when you have a prediction of what evidence should be found, may be evidence against. So you need two things there: 1) absence of evidence; and 2) a prediction of what evidence there should've been. If all you have is #1 by itself, you have no evidence of anything. Hence, absence of evidence (without more) isn't evidence of absence. Maybe I'm in the wrong debates, but most of the times when I see the slogan applied, the person doesn't have the second part. For example, a common elementary error is to conclude that variables are uncorrelated because you calculated their correlation in your data set and it was not significant. Failure to find a correlation in a very large population study that's large enough that it should have found one, can be evidence of no correlation (or no large correlation, at least), but you have to do a separate analysis for that.

smanek 16y ago

It's a pretty straightforward implication of bayes' law that the absence of evidence is evidence of absence. In fact, we can quite easily quantify just how strong the evidence it provides is. http://lesswrong.com/lw/ih/absence_of_evidence_is_evidence_of_absence/ http://lesswrong.com/lw/ih/absence_of_evidence_is_evidence_o...

PaulHoule 16y ago
But, with Bayesian methods you're feeding in prior information which can itself be flawed. When it comes to some question like 'What is reliability of the space shuttle?' the largest uncertainty in the calculation is that there is some failure mode you haven't thought of. Despite decades of changes to the space shuttle to improve safety, the official estimate of the failure probability hasn't changed since 1975. In conventional logic, we say that a reasoning method is sound if it never comes to correct conclusions. One version of 'A of E' -> 'E of A' is the "closed world assumption" which, in one variant, means that "if you haven't been presented with X, then not(X) is true." Now, we know that's not sound, not correct 100% of the time. However, when doing 'commonsense reasoning', the CWA can sometimes be a highly effective heuristic. (Note also that's there's no completely satisfactory way to deal with logical negation in knoweldge based systems at this time, so we've got to choose some poison, or otherwise deal with a system that "can't say no". Part of that is that 'commonsense reasoning' is probabilistic. For instance, an effective hold 'em player has to estimate the probability of improving his hand at drawing, have a guess about what his opponent(s) may be holding, and how the opponent may respond to his actions. From that he's able to estimate odds which help him decide what to bet. Poker's a pure example, but the same kind of probabilistic reasoning is necessary for hunting, software project management, scientific research, and many of things that people do.
amalcon 16y ago
That article is subtly wrong. Certainly it is less wrong (pun intended) than the reasoning it criticizes, but it does subtly miss the mark. Bayes' Law refers to data: specifically, it relates data to evidence. When a datum provides evidence for a hypothesis, the reverse of that datum must also provide evidence against that hypothesis. That is to say, a datum is either relevant or it's not. If it is relevant, then it will present evidence either for or against a given hypothesis, depending on its value. If you don't know that datum, however, it manages to not provide evidence either way. As such, it is easy to lack evidence either for or against a hypothesis. Through straightforward logical manipulations from there, it's possible to have "absence of evidence" while not having "evidence of absence". Therefore, they cannot be equivalent.

baddox 16y ago

The big point here, which the author doesn't mention, is that you can have absence of evidence without searching for evidence or you can have absence of evidence after rigorous searching. I believe that the former situation is the one covered by the saying "absence of evidence is not evidence of absence." For example, I personally haven't observed any evidence of gravitational time dilation, because I haven't looked. I would be a fool to interpret this lack of evidence as evidence that gravitational time dilation doesn't exist. The engineers of the Global Positioning System have evidence. However, after a reasonably rigorous search, one may indeed reasonably conclude that something doesn't exist if no evidence of it has been found, especially when evidence would be easily visible if the thing did exist. The examples the author gave fall into this category: Most likely, given the distribution of humans on Earth, there would be evidence of dodo birds if they were extant. Now, it's important to be careful when estimating the likelihood of stumbling upon evidence of something. Hundreds of new species (most only subtly different than known species) are discovered every year, and there are undoubtedly thousands of oceanic creatures that have never been seen by humans, much less captured and dissected. It would be foolish to deny that there aren't hundreds or thousands of yet-undiscovered species simply because as of now we have no evidence of them.

tjmaxal 16y ago

The turkey comment reminds me of black swan investing. Just because it hasn't happened yet doesn't mean it won't in fact for certain things, it actually increases the odds that it will happen the more time passes without it occurring, like Volcanic eruptions.

Silhouette 16y ago

Another slightly annoying saying is "The plural of anecdote is not data". Any anecdote is a data point, if you assume that it is factually correct. You may or may not be able to draw any significant conclusions from that data, once you take into account sample size and bias, of course. But unless you're going to "sample" the entire population of interest, pretty much all statistical analysis in the real world is done based on multiple anecdotes. We just try to minimise the bias and to choose a sample size large enough to give results significant enough for our purposes.

Panoramix 16y ago
I disagree. I interpret that phrase as stating that an anecdote is something fundamentally different from a valid data point: the data point came from a carefully designed experiment using properly tested equipment, control groups, rigorous mathematical analysis, perhaps using double-blind, randomized trials where it applies, etc... whereas the anecdote is a story you hear from somebody, as per its definition: http://en.wikipedia.org/wiki/Anecdotal_evidence http://en.wikipedia.org/wiki/Anecdotal_evidence
wglb 16y ago
Ah, but you see, there are those that say that "data" by definition is plural.