4 ms·

On the other hand when you apply any non mathematical (ie. not standard deviation, etc) segmentation on the dataset you are artificially fitting the data. The

On the other hand when you apply any non mathematical (ie. not standard deviation, etc) segmentation on the dataset you are artificially fitting the data. The correct approach would be to search for patterns over time in the dataset (frequency analysis) and see what turns up. not arbitrary segment the data into week sized blocks because you 'think' there might be a data pattern in there. I see this sort of 'inspiration based' analysis in web analytics all the time, and it's complete nonsense. Look for patterns in the data, don't look at the world and try to fit the data to it. You'll end up with stupid and statistically invalid results.

ekianjo 14y ago
I see your point but I do not fully agree with your view. There are so many times where we see patterns, correlations emerging from the data, which make no sense whatsoever. Statistical correlations can happen by chance (by definition) and something sticking out does not mean that something is happening. You would need to repeat the analysis on different pools of data to prove that there is indeed something you are missing. By looking at established segments, at least you know where to look for things that MAY make sense, and then derive hypothesis. If not, you end with bullcrap analysis we see every single day in literature where researchers find a "link between vegetarians and personality disorder" or things like that, without even probing the fact such links may be coincidental at best.
- shadowmint 14y ago
  The problem is that mathematically when you partition your data into sub-set and analyse those subsets you are no longer analysing the original data set. Your data is now composed of the original set of data [n1, n2, n3... nn] + a NEW DATA SET [s1, s2, s3 ... sn] which is your selection criteria. This is often masked by the fact that your selection data set is a function f(x) ("we'll just split them up by week and partition by gender"), so it doesn't look like your actually combining two dataset, but you are. This new set of data S, biases the result of the analysis, and the more complex it is, the closer you are to over-fitting to find the 'golden correlation' you're looking for in your data. Humans are pattern matching machines. We see patterns in random noise, and hear voices in random tones. It's easy for us to see patterns (or what we think are patterns) in data sets, but its much harder to statistically substantiate those patterns as distinct from randomness. It's provable, I imagine, that any data pattern can be found in a data set if we have sufficiently complex selection criteria for the samples, and sufficiently random raw data. dont do this If you do, at the very least acknowledge that you have selectively modified the original dataset and specifically test your partitioning rules. Its much, much better to search the dataset for a target pattern, and then investigate the partitioning rules that your search matches against (because you can examine the partitioning rules for complexity, and data-content and easily spot over-fitting).

ekianjo 14y ago

I see your point but I do not fully agree with your view. There are so many times where we see patterns, correlations emerging from the data, which make no sense whatsoever. Statistical correlations can happen by chance (by definition) and something sticking out does not mean that something is happening. You would need to repeat the analysis on different pools of data to prove that there is indeed something you are missing. By looking at established segments, at least you know where to look for things that MAY make sense, and then derive hypothesis. If not, you end with bullcrap analysis we see every single day in literature where researchers find a "link between vegetarians and personality disorder" or things like that, without even probing the fact such links may be coincidental at best.

shadowmint 14y ago
The problem is that mathematically when you partition your data into sub-set and analyse those subsets you are no longer analysing the original data set. Your data is now composed of the original set of data [n1, n2, n3... nn] + a NEW DATA SET [s1, s2, s3 ... sn] which is your selection criteria. This is often masked by the fact that your selection data set is a function f(x) ("we'll just split them up by week and partition by gender"), so it doesn't look like your actually combining two dataset, but you are. This new set of data S, biases the result of the analysis, and the more complex it is, the closer you are to over-fitting to find the 'golden correlation' you're looking for in your data. Humans are pattern matching machines. We see patterns in random noise, and hear voices in random tones. It's easy for us to see patterns (or what we think are patterns) in data sets, but its much harder to statistically substantiate those patterns as distinct from randomness. It's provable, I imagine, that any data pattern can be found in a data set if we have sufficiently complex selection criteria for the samples, and sufficiently random raw data. dont do this If you do, at the very least acknowledge that you have selectively modified the original dataset and specifically test your partitioning rules. Its much, much better to search the dataset for a target pattern, and then investigate the partitioning rules that your search matches against (because you can examine the partitioning rules for complexity, and data-content and easily spot over-fitting).