6 ms·

A Critical Field Guide for Working with Machine Learning Datasets

87 points by Amorymeltzer 4y ago

jna_sh 4y ago
I love the faux spreadsheet presentation of this guide.
- firatsarlar 4y ago
  +1
- r3trohack3r 4y ago
  I didn’t realize it wasn’t a “real” spreadsheet application until your comment. The design gave me a bit of anxiety, it was so convincing I was worried about clicking into one of the cells! Anxiety included - I also love the layout.
clircle 4y ago
100 years ago statisticians starting working on the theories of survey design and experimental design to clarify what types, how much, what mixture, and how to collect data. Machine learning people could learn a lot from these old, uncool theories if they want to approach data with rigor. That one can approach data collection and usage in a systematic way is totally lost now, it's just web scraping.
- steppi 4y ago
  This hasn’t been my experience. I don’t think this knowledge has been lost and it’s often still applied when incentivized, such as when quality output of machine learning models serves an immediate business need, or in scientific applications of machine learning. By your reference to web scraping I assume your referring to datasets for large language models, but I don’t think the kitchen sink approach is applied out of ignorance, but out of a principled decision to forgo data hygiene in order to collect datasets large enough to feed such large models. Also, strictly speaking, one only needs to apply the critical approach when constructing and using validation datasets, and it’s typical for larger noisier training datasets to outperform smaller more rigorous ones when models trained on both are tested on carefully constructed validation datasets.
- kordlessagain 4y ago
  A system for this would allow individual collections of documents with discussions attached. Without people, the machines won’t know where to put attention. That may change when they have individual identity, but for now the work probably needs to be done cooperatively.
- KRAKRISMOTT 4y ago
  Machine learning (or more specifically deep learning) is shaking the world precisely because the models at the highest end are able to take in mostly unlabeled garbage and spit out useful results in an unsupervised manner. The world is not full of i.i.d. spherical cows. 100 years of statistical advances have not been able to solve image recognition at scale until convolutional neural networks. Large language models may be somewhat Bayesian but they do not resemble any traditional Bayesian techniques except at the highest level. Clean data and good sampling are incompatible with the real world outside of laboratory conditions (or large budgets).
  - avs733 4y ago
    That is a fairly brave statement given some of the research looking at how failed attempts at ML modeling fall apart [1] [1] https://ai.googleblog.com/2021/06/data-cascades-in-machine-learning.html https://ai.googleblog.com/2021/06/data-cascades-in-machine-l...
  - wongarsu 4y ago
    Some of them do, but others depend on labeled and augmented data. For example training facial recognition starts with images labeled by who appears in them, then you run a model to find the face in the image and crop to that, then another image tells you where the eyes are and you use that to rotate the image to make the eyes level, and that's the input to the actual facial recognition neural network. And that's for the "naive" approaches that don't have special handling for faces don't look at the camera. And the field still routinely has problems because it turns out that somebody's data set only contained certain lighting conditions, or severely underrepresented certain races, etc.
    - KRAKRISMOTT 4y ago
      Not sure about image recognition but for the language models and audio space, semi/self-supervised methods are currently doing quite well.
- dachryn 4y ago
  you are too pessimistic. Senior profiles definitely dived into the statistics aspect of it all. Experiment design is often a big part of modern business cases, because naturally it is part of the cost to obtain data. It is still taught at university if you do a master degree or PhD at a decent place. But I agree that many juniors have no clue. Especially those who are self taught and join in from a different field
- razor_router 4y ago
  I definitely agree that taking a systematic approach to data collection and usage is incredibly important. However, I think it is impossible to simply dismiss the ways that modern machine learning has revolutionized the way we understand and interact with data. There is no question that rigorous survey design and experimental design theories still have merit and should be taken into account, but I think machine learning has allowed us to delve deeper into data analysis and uncover patterns and insights that would not be possible without its advancements.
kordlessagain 4y ago
Interesting it doesn’t cover technologies for storing data. It does mention analytical operations though. Anyone have a suggestion for storing large amounts of ML data, training sets or otherwise? I’ve been using FeatureBase and weaviate, and would be interested in learning about other solutions.
- kacperlukawski 4y ago
  If you want to perform vector search over your data, then Qdrant (https://qdrant.tech https://qdrant.tech) is worth checking out.
- gschoeni 4y ago
  We've been working on an open source tool called Oxen to help store large ML datasets. It's optimized for large sets of unstructured data ie images, video, audio, text, as well as parquet or arrow style DataFrames. Would love to get some feedback on it! https://github.com/Oxen-AI/oxen-release#-oxen https://github.com/Oxen-AI/oxen-release#-oxen
- wongarsu 4y ago
  "ML data" could be just about anything: images, sounds, graphs, tabular data, vectors. Depending on what your data is, you might end up with very different storage solutions. Though I think one common approach is to just dump most data in an S3-compatible datastore of your choice (there's seaweedfs or ceph, or the cloud provider of your choice). Specialized databases make a lot of sense when you need features like vector similarity search though.
- fzliu 4y ago
  Vector databases in general are good for storing large amounts of unstructured data by first converting them into embeddings via ML models. There's also feature stores, which store and organize features for later use in model training or predictive analytics. Feature stores generally come in _before_ models get trained, while vector databases generally come _after_ (i.e. they use trained models). Milvus (https://milvus.io https://milvus.io) and Feast (https://feast.dev/ https://feast.dev/) are two of the most well known vector databases and feature stores, respectively.
m00dy 4y ago
terrible layout to read..
peanutcrisis 4y ago
Note that the term "Critical" here refers to the likes of "Critical Theory", "Critical Social Justice", "Critical Race Theory" etc. It's biased from the offset so I'd take anything from this site with a pinch of salt.

87 points by Amorymeltzer 4y ago

jna_sh 4y ago

I love the faux spreadsheet presentation of this guide.

firatsarlar 4y ago
+1
r3trohack3r 4y ago
I didn’t realize it wasn’t a “real” spreadsheet application until your comment. The design gave me a bit of anxiety, it was so convincing I was worried about clicking into one of the cells! Anxiety included - I also love the layout.

clircle 4y ago

100 years ago statisticians starting working on the theories of survey design and experimental design to clarify what types, how much, what mixture, and how to collect data. Machine learning people could learn a lot from these old, uncool theories if they want to approach data with rigor. That one can approach data collection and usage in a systematic way is totally lost now, it's just web scraping.

steppi 4y ago
This hasn’t been my experience. I don’t think this knowledge has been lost and it’s often still applied when incentivized, such as when quality output of machine learning models serves an immediate business need, or in scientific applications of machine learning. By your reference to web scraping I assume your referring to datasets for large language models, but I don’t think the kitchen sink approach is applied out of ignorance, but out of a principled decision to forgo data hygiene in order to collect datasets large enough to feed such large models. Also, strictly speaking, one only needs to apply the critical approach when constructing and using validation datasets, and it’s typical for larger noisier training datasets to outperform smaller more rigorous ones when models trained on both are tested on carefully constructed validation datasets.
kordlessagain 4y ago
A system for this would allow individual collections of documents with discussions attached. Without people, the machines won’t know where to put attention. That may change when they have individual identity, but for now the work probably needs to be done cooperatively.
KRAKRISMOTT 4y ago
Machine learning (or more specifically deep learning) is shaking the world precisely because the models at the highest end are able to take in mostly unlabeled garbage and spit out useful results in an unsupervised manner. The world is not full of i.i.d. spherical cows. 100 years of statistical advances have not been able to solve image recognition at scale until convolutional neural networks. Large language models may be somewhat Bayesian but they do not resemble any traditional Bayesian techniques except at the highest level. Clean data and good sampling are incompatible with the real world outside of laboratory conditions (or large budgets).
- avs733 4y ago
  That is a fairly brave statement given some of the research looking at how failed attempts at ML modeling fall apart [1] [1] https://ai.googleblog.com/2021/06/data-cascades-in-machine-learning.html https://ai.googleblog.com/2021/06/data-cascades-in-machine-l...
- wongarsu 4y ago
  Some of them do, but others depend on labeled and augmented data. For example training facial recognition starts with images labeled by who appears in them, then you run a model to find the face in the image and crop to that, then another image tells you where the eyes are and you use that to rotate the image to make the eyes level, and that's the input to the actual facial recognition neural network. And that's for the "naive" approaches that don't have special handling for faces don't look at the camera. And the field still routinely has problems because it turns out that somebody's data set only contained certain lighting conditions, or severely underrepresented certain races, etc.
  - KRAKRISMOTT 4y ago
    Not sure about image recognition but for the language models and audio space, semi/self-supervised methods are currently doing quite well.
dachryn 4y ago
you are too pessimistic. Senior profiles definitely dived into the statistics aspect of it all. Experiment design is often a big part of modern business cases, because naturally it is part of the cost to obtain data. It is still taught at university if you do a master degree or PhD at a decent place. But I agree that many juniors have no clue. Especially those who are self taught and join in from a different field
razor_router 4y ago
I definitely agree that taking a systematic approach to data collection and usage is incredibly important. However, I think it is impossible to simply dismiss the ways that modern machine learning has revolutionized the way we understand and interact with data. There is no question that rigorous survey design and experimental design theories still have merit and should be taken into account, but I think machine learning has allowed us to delve deeper into data analysis and uncover patterns and insights that would not be possible without its advancements.

kordlessagain 4y ago

Interesting it doesn’t cover technologies for storing data. It does mention analytical operations though. Anyone have a suggestion for storing large amounts of ML data, training sets or otherwise? I’ve been using FeatureBase and weaviate, and would be interested in learning about other solutions.

kacperlukawski 4y ago
If you want to perform vector search over your data, then Qdrant (https://qdrant.tech https://qdrant.tech) is worth checking out.
gschoeni 4y ago
We've been working on an open source tool called Oxen to help store large ML datasets. It's optimized for large sets of unstructured data ie images, video, audio, text, as well as parquet or arrow style DataFrames. Would love to get some feedback on it! https://github.com/Oxen-AI/oxen-release#-oxen https://github.com/Oxen-AI/oxen-release#-oxen
wongarsu 4y ago
"ML data" could be just about anything: images, sounds, graphs, tabular data, vectors. Depending on what your data is, you might end up with very different storage solutions. Though I think one common approach is to just dump most data in an S3-compatible datastore of your choice (there's seaweedfs or ceph, or the cloud provider of your choice). Specialized databases make a lot of sense when you need features like vector similarity search though.
fzliu 4y ago
Vector databases in general are good for storing large amounts of unstructured data by first converting them into embeddings via ML models. There's also feature stores, which store and organize features for later use in model training or predictive analytics. Feature stores generally come in _before_ models get trained, while vector databases generally come _after_ (i.e. they use trained models). Milvus (https://milvus.io https://milvus.io) and Feast (https://feast.dev/ https://feast.dev/) are two of the most well known vector databases and feature stores, respectively.

m00dy 4y ago

terrible layout to read..

peanutcrisis 4y ago

Note that the term "Critical" here refers to the likes of "Critical Theory", "Critical Social Justice", "Critical Race Theory" etc. It's biased from the offset so I'd take anything from this site with a pinch of salt.