2 ms·
I would argue it's more scientific (even though the goal generally isn't science). Hypothesize, design measurements, test, conclude. Modern science often has t
by dalbasal 8y ago
I would argue it's more scientific (even though the goal generally isn't science). Hypothesize, design measurements, test, conclude.
Modern science often has the opposite issue. Economists (for example) have a lot of data. They can automate regression analysis to find correlations, then fit the theory (in economics, this is often an intricate model of an economic system) to the "result."
The problem is, 99% certainty doesn't mean anything if effectively test 1000 hypotheses. 10 correlations will occur randomly.
- jayd16 8y agoMore data does not make you a worse data scientist. You can run a bad study at any amount of data. The only difference is a new hypothesis will have historical data to look at. More data not require you create some over fitted model. More data is strictly better.
- edmundsauto 8y agoI disagree with "More data is strictly better". if you said that everything else equal, more data is better, I would be closer to agreement. But when you have more data, biases become more of a problem. p-hacking becomes easier, and the easier something is to do, the more likely it is to happen. I would frame it this way: the signal/noise ratio of data decreases as the size increases.* The overall value increases, but at a slower rate over time. * The caveat - once you get above a certain volume of data, new processing techniques become available that aren't available at smaller volumes.