4 ms·

A powerful concept for thinking about small data is bias/variance tradeoff. The ELI5 is: data is how a learner selects confidently between one model and anothe

A powerful concept for thinking about small data is bias/variance tradeoff. The ELI5 is: data is how a learner selects confidently between one model and another. So if you have little data, you must choose between high bias (fewer models to choose from) or low confidence (can't be sure you picked the best one). Bias in a lot of contexts is Bad, and large data techniques like DNN are basically about how you can have biases that are so vague the learner has plenty of room to surprise you. But when you add bias that truly aligns with the structure your learner is going to be exposed to, you enable it to reach confidence sooner. Many important techniques for small data are about allowing you to express specific biases. For example: - PGMs and Probabilistic Programming are about giving you a specific, interpretable structure for how your data are related to each other. - Picking informed bayesian priors - Data augmentation lets you add bias by expressing what kinds of variation don't matter (I.e. permitting and fuzzing your dataset) - feature engineering is about selectively adding and removing biases in terms of what kinds of data transformations are informative With all that said, the large data ML community has produced an incredible tool for practical work with small data: transfer learning. If your dataset is related to a larger one (e.g. by including natural language), you can borrow informed biases from models that were trained on a much larger corpus.

Hendrikto 4y ago
> Data augmentation lets you add bias by expressing what kinds of variation don't matter (I.e. permitting and fuzzing your dataset) What does ”permitting your dataset“ mean? Is it a typo? Did you mean ”permuting“?
- evrydayhustling 4y ago
  Yes, that was supposed to be "permuting"!
xksteven 4y ago
I'd be careful of over applying the "bias-variance tradeoff." How to define the variance of a model is not a simple task. I wouldn't say it is immediately obvious how bias-variance relates to small data scenarios. How much data is considered small? What is the complexity of the dataset itself? Even in Machine Learning it is possible to learn from small datasets without transfer learning. See meta-learning for instance.
- evrydayhustling 4y ago
  > I'd be careful of over applying the "bias-variance tradeoff." How to define the variance of a model is not a simple task. I wouldn't say it is immediately obvious how bias-variance relates to small data scenarios. It's very important for anyone studying ML to understand how bias-variance relates to sample size, so I'd encourage finding more resources if this note didn't help clarify! Here's another shot at a summary: for a fixed size of training sample, you must trade off between sensitivity to randomness in the sample (variance) and assumptions that bias the model you train. It's true that quantifying variance and bias can be hard, and you need systems like PAC learning to go further and actually estimate sufficient sample sizes for a task. But you can still reason usefully about any system that involves using data to select (train) among a class of potential output models! For example, the statement about meta-learning is incorrect, at least as far as i've seen the term used. Meta learning involves learning hyperparameters (including functions) that are then used to train a model. The extra stage makes these models less biased, but actually require more data. (Of course, in some meta learning systems, hyper parameters are learned with the help of external data - a form of transfer learning.)

Hendrikto 4y ago

> Data augmentation lets you add bias by expressing what kinds of variation don't matter (I.e. permitting and fuzzing your dataset) What does ”permitting your dataset“ mean? Is it a typo? Did you mean ”permuting“?

evrydayhustling 4y ago
Yes, that was supposed to be "permuting"!

xksteven 4y ago

I'd be careful of over applying the "bias-variance tradeoff." How to define the variance of a model is not a simple task. I wouldn't say it is immediately obvious how bias-variance relates to small data scenarios. How much data is considered small? What is the complexity of the dataset itself? Even in Machine Learning it is possible to learn from small datasets without transfer learning. See meta-learning for instance.

evrydayhustling 4y ago
> I'd be careful of over applying the "bias-variance tradeoff." How to define the variance of a model is not a simple task. I wouldn't say it is immediately obvious how bias-variance relates to small data scenarios. It's very important for anyone studying ML to understand how bias-variance relates to sample size, so I'd encourage finding more resources if this note didn't help clarify! Here's another shot at a summary: for a fixed size of training sample, you must trade off between sensitivity to randomness in the sample (variance) and assumptions that bias the model you train. It's true that quantifying variance and bias can be hard, and you need systems like PAC learning to go further and actually estimate sufficient sample sizes for a task. But you can still reason usefully about any system that involves using data to select (train) among a class of potential output models! For example, the statement about meta-learning is incorrect, at least as far as i've seen the term used. Meta learning involves learning hyperparameters (including functions) that are then used to train a model. The extra stage makes these models less biased, but actually require more data. (Of course, in some meta learning systems, hyper parameters are learned with the help of external data - a form of transfer learning.)