4 ms·

I think "big" data is being confused with "ubiquitous" data in this article. Larger datasets will always lead to more statistical confidence in a conclusion you

by rm999 10y ago

I think "big" data is being confused with "ubiquitous" data in this article. Larger datasets will always lead to more statistical confidence in a conclusion you make from the data. The article does a great job of explaining the caveats to this (data skews, misuse of stats, external factors), but those issues exist with small data too. In other words, this isn't about the volume of data. I think there are two real and different issues at play here, and both come instead from the ubiquitousness of data nowadays: 1. It's become easier to do more experiments, so even experts are more likely to produce some bad conclusions. 2. Data has become much more accessible, so people without rigorous stats backgrounds have an easier time abusing the hell out of stats on datasets.

deleted 10y ago
[deleted]
skybrian 10y ago
Perhaps a better way to put it is that fixing biased sampling is more useful than increasing the size of the dataset. If I'm profiling a program using a sampling profiler and it's biased, running it longer to collect more data is easy but doesn't help much - it's still biased but with higher resolution. If you can somehow adjust to remove the bias more resolution might still be useful but you can't adjust for data that's not there - suppose the profiler never samples during garbage collection. Another example is photography where a longer exposure (more photons) can be useful but won't fix fundamental problems with the camera or how the photo was taken. If the camera is out of focus or shakes because you don't have a tripod or image stabilization, a longer exposure won't fix it and might make it worse.
Fiahil 10y ago
I can't agree more with your second point, given the time I spent trying to explain various "disappointing" results to digital marketing people.
Obi_Juan_Kenobi 10y ago
> Larger datasets will always lead to more statistical confidence I think I understand what you're trying to say, but I don't agree with how you've phrased it. More data can lower p-values or increase power for a given analysis, but the assumptions that go into using data don't change when you simply have more of it. And those assumptions are everything in statistics. In fact, I think the temptation of 'more is better' leads to more use of what is easily available which can be highly biased. You also get much more 'significant' results that are more tempting to believe in. It's harder to shy away from a very low p-value, even when you know the sampling may not be appropriate. I'm a biologist, and as a field we have a lot of issues to address with bioinformatics. Over-eager investigators will pull out interesting tidbits from datasets without considering how problematic that sort of hypothesis generation can be. Fortunately the grant reviewers seem to be well aware of this for the most part. I've heard of many people having grants rejected because they thought they found a 'one in a million' phenomenon, but it turns out they looked a million times to find it :) Good bioinformatics is still firmly grounded in genetics.
- rm999 10y ago
  Your comment is mostly addressing what I refer to as "data skews, misuse of stats, and external factors", and is what the article is about. Yes, those are huge issues, but they have little to do with data size. I'm confused why you don't agree with my phrasing. To be clear, and from your comment I think you already know this, p-value is not a measure of confidence. P-value is a terrible metric that jumbles confidence with effect size. As you get more data you have the ability to properly detect small effect sizes with confidence; hence, the part of my comment you quoted and seem to disagree with? But people who don't have experience working with large datasets see a low p-value and often think they have a big effect size with reasonable confidence instead of a small effect size with very high confidence. Chalk this up to a "misuse of stats" :)

by rm999 10y ago

deleted 10y ago

[deleted]

skybrian 10y ago

Perhaps a better way to put it is that fixing biased sampling is more useful than increasing the size of the dataset. If I'm profiling a program using a sampling profiler and it's biased, running it longer to collect more data is easy but doesn't help much - it's still biased but with higher resolution. If you can somehow adjust to remove the bias more resolution might still be useful but you can't adjust for data that's not there - suppose the profiler never samples during garbage collection. Another example is photography where a longer exposure (more photons) can be useful but won't fix fundamental problems with the camera or how the photo was taken. If the camera is out of focus or shakes because you don't have a tripod or image stabilization, a longer exposure won't fix it and might make it worse.

Fiahil 10y ago

I can't agree more with your second point, given the time I spent trying to explain various "disappointing" results to digital marketing people.

Obi_Juan_Kenobi 10y ago

> Larger datasets will always lead to more statistical confidence I think I understand what you're trying to say, but I don't agree with how you've phrased it. More data can lower p-values or increase power for a given analysis, but the assumptions that go into using data don't change when you simply have more of it. And those assumptions are everything in statistics. In fact, I think the temptation of 'more is better' leads to more use of what is easily available which can be highly biased. You also get much more 'significant' results that are more tempting to believe in. It's harder to shy away from a very low p-value, even when you know the sampling may not be appropriate. I'm a biologist, and as a field we have a lot of issues to address with bioinformatics. Over-eager investigators will pull out interesting tidbits from datasets without considering how problematic that sort of hypothesis generation can be. Fortunately the grant reviewers seem to be well aware of this for the most part. I've heard of many people having grants rejected because they thought they found a 'one in a million' phenomenon, but it turns out they looked a million times to find it :) Good bioinformatics is still firmly grounded in genetics.

rm999 10y ago
Your comment is mostly addressing what I refer to as "data skews, misuse of stats, and external factors", and is what the article is about. Yes, those are huge issues, but they have little to do with data size. I'm confused why you don't agree with my phrasing. To be clear, and from your comment I think you already know this, p-value is not a measure of confidence. P-value is a terrible metric that jumbles confidence with effect size. As you get more data you have the ability to properly detect small effect sizes with confidence; hence, the part of my comment you quoted and seem to disagree with? But people who don't have experience working with large datasets see a low p-value and often think they have a big effect size with reasonable confidence instead of a small effect size with very high confidence. Chalk this up to a "misuse of stats" :)