4 ms·

To be really automatic the only thing i should need to do is feed an csv, correct the suggested data types and then run all algorithms on the data with the info

by tpetry 6y ago

To be really automatic the only thing i should need to do is feed an csv, correct the suggested data types and then run all algorithms on the data with the information at the end which has been the most effective and which i should further optimize.

nidhaloff 6y ago
this is a great feature! Thanks for the feedback
- fractionalhare 6y ago
  No, it's emphatically not a great feature, and it's not clear to me the commenter was recommending that so much as making a nit. Please don't automate the process of choosing and running algorithms on a single sample of data, it's unsound experimental design that undermines your results. If you insist on doing it anyway, at minimum you will need to automate an initial assessment of the sample data to determine if it has a suitable size and distribution to allow you to adjust the significance of results for the number of tests you're running, and partition the data into smaller subsamples.
  - nidhaloff 6y ago
    Hi, thanks for your comment. I actually understood that he meant something like a hyperparameter search/tuning using cross validation (at least that what came in my mind).
    - fractionalhare 6y ago
      Cross validation would be good! I think if you build this in you could automatically run a few heuristics to see if the data can be partitioned, or maybe just prompt the user for another sample of the data with the same distribution.
    - tpetry 6y ago
      Parameter tuning and algorithm selection! I just don’t want to manually start 5 different runs of algorithms i believe which could work good on the data and manually compare the results. And maybe i was too lazy to run the 6th algorithm which now performs much better. But to be sure, every test should be done with k-fold cross validation. The decision whether to split the training set should not be chosen by the user. It‘s crucial that this is a must!
fractionalhare 6y ago
Sure, but shotgunning every statistical test and machine learning algorithm completely undermines your results, because the power and significance levels are not adjusted for many experiments. In the statistical setting this leads to spurious correlations, and in the machine learning setting it leads to overfitting. In either case the results have a high risk of not generalizing beyond the initial sample being analyzed. I'm not saying you're endorsing this, but it's basically antithetical to sound experimental design. I don't think the author should pursue automatic anything when it comes to statistics, unless it's just a thin quality-of-life wrapper around other statistical primitives and libraries.
- streetcat1 6y ago
  So over-fitting can be solved by : 1) Using cross validation/validation set. 2) Regularisation . 3) Finding statistically significant features (e.g. chi square). Why cant this be done automatically? I.e. what is the human advantage ?
  - fractionalhare 6y ago
    It's not that you intrinsically need a human, it's that doing this without human oversight requires being very careful not to make tricky mistakes. The nature of statistical significance (which underpins everything you've said), is that repeating many experiments reduces the confidence you should have in your results. Supposing each algorithm is an experiment and each experiment is independent, if you target a significance level of p = 0.05, you can expect to find 1 correlated feature out of every 20 you test just by chance. Can you automatically correct for this? Sure. But this is just one possible footgun. Are you confident you're avoiding them all? In theory automation could do an even better job than a human of avoiding the myriad statistical mistakes you could make, but in practice that requires significant upfront effort and expertise during the development process. At a certain point doing this automatically becomes analogous to rolling your own crypto. It's not quite an adversarial problem, but it's quite easy to screw up. I agree that cross validating would work; that's what I was gesturing to when I was talking about making an assessment of the data and partitioning it. Either the provided sample should be partitioned for cross validation, or it should prompt the user for a second set.

by tpetry 6y ago

nidhaloff 6y ago

this is a great feature! Thanks for the feedback

fractionalhare 6y ago
No, it's emphatically not a great feature, and it's not clear to me the commenter was recommending that so much as making a nit. Please don't automate the process of choosing and running algorithms on a single sample of data, it's unsound experimental design that undermines your results. If you insist on doing it anyway, at minimum you will need to automate an initial assessment of the sample data to determine if it has a suitable size and distribution to allow you to adjust the significance of results for the number of tests you're running, and partition the data into smaller subsamples.
- nidhaloff 6y ago
  Hi, thanks for your comment. I actually understood that he meant something like a hyperparameter search/tuning using cross validation (at least that what came in my mind).
  - fractionalhare 6y ago
    Cross validation would be good! I think if you build this in you could automatically run a few heuristics to see if the data can be partitioned, or maybe just prompt the user for another sample of the data with the same distribution.
  - tpetry 6y ago
    Parameter tuning and algorithm selection! I just don’t want to manually start 5 different runs of algorithms i believe which could work good on the data and manually compare the results. And maybe i was too lazy to run the 6th algorithm which now performs much better. But to be sure, every test should be done with k-fold cross validation. The decision whether to split the training set should not be chosen by the user. It‘s crucial that this is a must!

fractionalhare 6y ago

Sure, but shotgunning every statistical test and machine learning algorithm completely undermines your results, because the power and significance levels are not adjusted for many experiments. In the statistical setting this leads to spurious correlations, and in the machine learning setting it leads to overfitting. In either case the results have a high risk of not generalizing beyond the initial sample being analyzed. I'm not saying you're endorsing this, but it's basically antithetical to sound experimental design. I don't think the author should pursue automatic anything when it comes to statistics, unless it's just a thin quality-of-life wrapper around other statistical primitives and libraries.

streetcat1 6y ago
So over-fitting can be solved by : 1) Using cross validation/validation set. 2) Regularisation . 3) Finding statistically significant features (e.g. chi square). Why cant this be done automatically? I.e. what is the human advantage ?
- fractionalhare 6y ago
  It's not that you intrinsically need a human, it's that doing this without human oversight requires being very careful not to make tricky mistakes. The nature of statistical significance (which underpins everything you've said), is that repeating many experiments reduces the confidence you should have in your results. Supposing each algorithm is an experiment and each experiment is independent, if you target a significance level of p = 0.05, you can expect to find 1 correlated feature out of every 20 you test just by chance. Can you automatically correct for this? Sure. But this is just one possible footgun. Are you confident you're avoiding them all? In theory automation could do an even better job than a human of avoiding the myriad statistical mistakes you could make, but in practice that requires significant upfront effort and expertise during the development process. At a certain point doing this automatically becomes analogous to rolling your own crypto. It's not quite an adversarial problem, but it's quite easy to screw up. I agree that cross validating would work; that's what I was gesturing to when I was talking about making an assessment of the data and partitioning it. Either the provided sample should be partitioned for cross validation, or it should prompt the user for a second set.