4 ms·

I'm a statistician, not a machine learning person, but >Small data sets are not appropriate for machine learning as they lead to high bias in their results (Va

I'm a statistician, not a machine learning person, but >Small data sets are not appropriate for machine learning as they lead to high bias in their results (Vabalas et al., 2019). To address this, we created a synthetic set data with 10,000 observations using the synthpop package in R (Nowok et al., 2016). This standard automated procedure creates observations by repeatedly randomly sampling the joint distribution of the data. This technique is used when obtaining large datasets is infeasible, including analyses of computer vision (Mayer et al., 2018), sensitive information like hospital records (Tucker et al., 2020), and with unbalanced data (He et al., 2008; Luo et al., 2018). Isn't this just making up data? How is the analysis of the synthetic 10,000 observations (based on observations from 33 real people) at all meaningful?

scotty79 3y ago
Curious, if it manages to classify real data correctly after such learning.
beerpls 3y ago
Yeah, this is an issue with way more science than anyone (esp scientists) is comfortable acknowledging In grad school I encountered experiment after experiment, paper after paper where the authors went around emphasizing the parts which were statistically “sexy” while ignoring glaring issues like this It’s almost like the methodology for creating science is more rigorous than the science itself Sometimes the sample is too small. Sometimes they assume a correlation that doesn’t exist. It’s troubling and was a large part of why I left academia
- CrazyStat 3y ago
  I faced similar pressures in graduate school to leave out somewhat unfavorable results (which I thought were interesting and worth presenting even if they didn't flatter our approach). It wasn't the only thing that drove me away from academia, but it certainly contributed. In the statistics department, of all places! Where the importance of being forthcoming with your data should be best understood!
mjburgess 3y ago
It's not. Indeed, "predictive accuracy" on simulated data is largely a meaningless metric in any case. The simulation's set up determines the predictive accuracy of any model of it. The only value of this metric, post-simulation, is to determine whether or not an experiment is worth the ROI -- ie., it's essentially a way of determining whether some set of premises (simulated) are possibly consistent with a given predictive project.
j16sdiz 3y ago
In machine learning, it is very common to _train_ with synthetic data. This is ok as long as them can evaluate with real, observed data. They included a discussion in appendix, but I cannot open that file.
- CrazyStat 3y ago
  I looked at the appendix but it's mostly just tables and density plots comparing the real and synthetic datasets. There's no additional justification of why it makes sense to use the synthetic dataset in the first place. As a statistician it still feels very weird to train on synthetic data, because my evaluation of model uncertainty (parameter uncertainty and predictive uncertainty) is (usually) based on the data the model is trained on. But I understand that ML typically evaluates models based on out-of-sample data, so training on synthetic data makes sense in that case. In this paper it appears they're also evaluating on the synthetic data, which seems problematic: > One-half of the synthetic data was used to train the bagged ML model and tune the hyperparameters. The other half of the synthetic data was used to test it.
  - moss2 3y ago
    Evaluating on synthetic data is... I want to say "forbidden" but that's not the right word. Basically all of their results are scientifically useless. I can't figure out which journal or conference this article was published to, but if I sat on the reviewer board I would have declined their article on this fact alone. Good find.
mensetmanusman 3y ago
Bootstrapping https://en.m.wikipedia.org/wiki/Bootstrapping_(statistics) https://en.m.wikipedia.org/wiki/Bootstrapping_(statistics) https://en.m.wikipedia.org/wiki/Bootstrap_aggregating https://en.m.wikipedia.org/wiki/Bootstrap_aggregating
- CrazyStat 3y ago
  I'm familiar with bootstrapping--both the classical bootstrap, which fits the model repeatedly on different simulated datasets to evaluate model uncertainty, and bagging, which fits the model repeatedly on different simulated datasets as inputs for ensemble learning. Neither of those is what was done here, which was fitting the model once on a greatly inflated synthetic sample.

scotty79 3y ago

Curious, if it manages to classify real data correctly after such learning.

beerpls 3y ago

Yeah, this is an issue with way more science than anyone (esp scientists) is comfortable acknowledging In grad school I encountered experiment after experiment, paper after paper where the authors went around emphasizing the parts which were statistically “sexy” while ignoring glaring issues like this It’s almost like the methodology for creating science is more rigorous than the science itself Sometimes the sample is too small. Sometimes they assume a correlation that doesn’t exist. It’s troubling and was a large part of why I left academia

CrazyStat 3y ago
I faced similar pressures in graduate school to leave out somewhat unfavorable results (which I thought were interesting and worth presenting even if they didn't flatter our approach). It wasn't the only thing that drove me away from academia, but it certainly contributed. In the statistics department, of all places! Where the importance of being forthcoming with your data should be best understood!

mjburgess 3y ago

It's not. Indeed, "predictive accuracy" on simulated data is largely a meaningless metric in any case. The simulation's set up determines the predictive accuracy of any model of it. The only value of this metric, post-simulation, is to determine whether or not an experiment is worth the ROI -- ie., it's essentially a way of determining whether some set of premises (simulated) are possibly consistent with a given predictive project.

j16sdiz 3y ago

In machine learning, it is very common to _train_ with synthetic data. This is ok as long as them can evaluate with real, observed data. They included a discussion in appendix, but I cannot open that file.

CrazyStat 3y ago
I looked at the appendix but it's mostly just tables and density plots comparing the real and synthetic datasets. There's no additional justification of why it makes sense to use the synthetic dataset in the first place. As a statistician it still feels very weird to train on synthetic data, because my evaluation of model uncertainty (parameter uncertainty and predictive uncertainty) is (usually) based on the data the model is trained on. But I understand that ML typically evaluates models based on out-of-sample data, so training on synthetic data makes sense in that case. In this paper it appears they're also evaluating on the synthetic data, which seems problematic: > One-half of the synthetic data was used to train the bagged ML model and tune the hyperparameters. The other half of the synthetic data was used to test it.
- moss2 3y ago
  Evaluating on synthetic data is... I want to say "forbidden" but that's not the right word. Basically all of their results are scientifically useless. I can't figure out which journal or conference this article was published to, but if I sat on the reviewer board I would have declined their article on this fact alone. Good find.

mensetmanusman 3y ago

Bootstrapping https://en.m.wikipedia.org/wiki/Bootstrapping_(statistics) https://en.m.wikipedia.org/wiki/Bootstrapping_(statistics) https://en.m.wikipedia.org/wiki/Bootstrap_aggregating https://en.m.wikipedia.org/wiki/Bootstrap_aggregating

CrazyStat 3y ago
I'm familiar with bootstrapping--both the classical bootstrap, which fits the model repeatedly on different simulated datasets to evaluate model uncertainty, and bagging, which fits the model repeatedly on different simulated datasets as inputs for ensemble learning. Neither of those is what was done here, which was fitting the model once on a greatly inflated synthetic sample.