5 ms·

100 years ago statisticians starting working on the theories of survey design and experimental design to clarify what types, how much, what mixture, and how to

100 years ago statisticians starting working on the theories of survey design and experimental design to clarify what types, how much, what mixture, and how to collect data. Machine learning people could learn a lot from these old, uncool theories if they want to approach data with rigor. That one can approach data collection and usage in a systematic way is totally lost now, it's just web scraping.

steppi 4y ago
This hasn’t been my experience. I don’t think this knowledge has been lost and it’s often still applied when incentivized, such as when quality output of machine learning models serves an immediate business need, or in scientific applications of machine learning. By your reference to web scraping I assume your referring to datasets for large language models, but I don’t think the kitchen sink approach is applied out of ignorance, but out of a principled decision to forgo data hygiene in order to collect datasets large enough to feed such large models. Also, strictly speaking, one only needs to apply the critical approach when constructing and using validation datasets, and it’s typical for larger noisier training datasets to outperform smaller more rigorous ones when models trained on both are tested on carefully constructed validation datasets.
kordlessagain 4y ago
A system for this would allow individual collections of documents with discussions attached. Without people, the machines won’t know where to put attention. That may change when they have individual identity, but for now the work probably needs to be done cooperatively.
KRAKRISMOTT 4y ago
Machine learning (or more specifically deep learning) is shaking the world precisely because the models at the highest end are able to take in mostly unlabeled garbage and spit out useful results in an unsupervised manner. The world is not full of i.i.d. spherical cows. 100 years of statistical advances have not been able to solve image recognition at scale until convolutional neural networks. Large language models may be somewhat Bayesian but they do not resemble any traditional Bayesian techniques except at the highest level. Clean data and good sampling are incompatible with the real world outside of laboratory conditions (or large budgets).
- avs733 4y ago
  That is a fairly brave statement given some of the research looking at how failed attempts at ML modeling fall apart [1] [1] https://ai.googleblog.com/2021/06/data-cascades-in-machine-learning.html https://ai.googleblog.com/2021/06/data-cascades-in-machine-l...
- wongarsu 4y ago
  Some of them do, but others depend on labeled and augmented data. For example training facial recognition starts with images labeled by who appears in them, then you run a model to find the face in the image and crop to that, then another image tells you where the eyes are and you use that to rotate the image to make the eyes level, and that's the input to the actual facial recognition neural network. And that's for the "naive" approaches that don't have special handling for faces don't look at the camera. And the field still routinely has problems because it turns out that somebody's data set only contained certain lighting conditions, or severely underrepresented certain races, etc.
  - KRAKRISMOTT 4y ago
    Not sure about image recognition but for the language models and audio space, semi/self-supervised methods are currently doing quite well.
dachryn 4y ago
you are too pessimistic. Senior profiles definitely dived into the statistics aspect of it all. Experiment design is often a big part of modern business cases, because naturally it is part of the cost to obtain data. It is still taught at university if you do a master degree or PhD at a decent place. But I agree that many juniors have no clue. Especially those who are self taught and join in from a different field
razor_router 4y ago
I definitely agree that taking a systematic approach to data collection and usage is incredibly important. However, I think it is impossible to simply dismiss the ways that modern machine learning has revolutionized the way we understand and interact with data. There is no question that rigorous survey design and experimental design theories still have merit and should be taken into account, but I think machine learning has allowed us to delve deeper into data analysis and uncover patterns and insights that would not be possible without its advancements.

steppi 4y ago

This hasn’t been my experience. I don’t think this knowledge has been lost and it’s often still applied when incentivized, such as when quality output of machine learning models serves an immediate business need, or in scientific applications of machine learning. By your reference to web scraping I assume your referring to datasets for large language models, but I don’t think the kitchen sink approach is applied out of ignorance, but out of a principled decision to forgo data hygiene in order to collect datasets large enough to feed such large models. Also, strictly speaking, one only needs to apply the critical approach when constructing and using validation datasets, and it’s typical for larger noisier training datasets to outperform smaller more rigorous ones when models trained on both are tested on carefully constructed validation datasets.

kordlessagain 4y ago

A system for this would allow individual collections of documents with discussions attached. Without people, the machines won’t know where to put attention. That may change when they have individual identity, but for now the work probably needs to be done cooperatively.

KRAKRISMOTT 4y ago

Machine learning (or more specifically deep learning) is shaking the world precisely because the models at the highest end are able to take in mostly unlabeled garbage and spit out useful results in an unsupervised manner. The world is not full of i.i.d. spherical cows. 100 years of statistical advances have not been able to solve image recognition at scale until convolutional neural networks. Large language models may be somewhat Bayesian but they do not resemble any traditional Bayesian techniques except at the highest level. Clean data and good sampling are incompatible with the real world outside of laboratory conditions (or large budgets).

avs733 4y ago
That is a fairly brave statement given some of the research looking at how failed attempts at ML modeling fall apart [1] [1] https://ai.googleblog.com/2021/06/data-cascades-in-machine-learning.html https://ai.googleblog.com/2021/06/data-cascades-in-machine-l...
wongarsu 4y ago
Some of them do, but others depend on labeled and augmented data. For example training facial recognition starts with images labeled by who appears in them, then you run a model to find the face in the image and crop to that, then another image tells you where the eyes are and you use that to rotate the image to make the eyes level, and that's the input to the actual facial recognition neural network. And that's for the "naive" approaches that don't have special handling for faces don't look at the camera. And the field still routinely has problems because it turns out that somebody's data set only contained certain lighting conditions, or severely underrepresented certain races, etc.
- KRAKRISMOTT 4y ago
  Not sure about image recognition but for the language models and audio space, semi/self-supervised methods are currently doing quite well.

dachryn 4y ago

you are too pessimistic. Senior profiles definitely dived into the statistics aspect of it all. Experiment design is often a big part of modern business cases, because naturally it is part of the cost to obtain data. It is still taught at university if you do a master degree or PhD at a decent place. But I agree that many juniors have no clue. Especially those who are self taught and join in from a different field

razor_router 4y ago

I definitely agree that taking a systematic approach to data collection and usage is incredibly important. However, I think it is impossible to simply dismiss the ways that modern machine learning has revolutionized the way we understand and interact with data. There is no question that rigorous survey design and experimental design theories still have merit and should be taken into account, but I think machine learning has allowed us to delve deeper into data analysis and uncover patterns and insights that would not be possible without its advancements.