46 ms·
At the root of this is something I'm constantly saying to other members in my research lab, arguing with reviewers, and passionately teaching my students: data
by godelski 3y ago
At the root of this is something I'm constantly saying to other members in my research lab, arguing with reviewers, and passionately teaching my students:
datasets are proxies, measurements are fuzzy.
This is something that is drilled into students learning statistics and I can't for the life of me figure out why this changed with respect to ML (best guess is not requiring statistics courses).
Datasets are proxies: they represent the world, but aren't. They should generally be seen as narrow subsets too. Your dataset quality and type matter a lot! Things like medical image datasets also have tons of correlating factors that can easily invalidate all results without you being aware. There are simple datasets we use to prove a concept (toys, mnist, cifar, etc). There are large scale datasets that have internal inconsistencies (imagenet, flowers). There are huge datasets that haven't been properly filtered/deduplicated (LAION). (there are also just shitty datasets (HumanEval)) Thinking of datasets as proxies helps internalize the frustration that literally every production engineer faces (even outside ML and software): real world results are inconsistent with lab results. Dataset engineering is an underappreciated art that is extremely difficult. But everyone needs to internalize that datasets are just a map, not the territory, and your navigation will only be as good as the map (many are poorly drawn maps, often on purpose)
Measurements are fuzzy: Benchmarkism is running rampant in the ML world and it baffles me that a field who's bottomline objective deals with alignment isn't able to align how we evaluate ourselves.
No measurement is perfect, many are far from them. You can train two language models to the same NLL and one might sample well and the other outputs garbage. You can train two image models to have identical FIDs and one samples clearly and the other doesn't. Likelihood also doesn't guarantee sharpness and I can go on. You must think about the limitations of your measurements and know them in depth. This also has seem to have gotten away from us and people just use the measurement libraries and call it that. We've reached a point where ImageNet classification accuracy has decoupled from downstream performance (object detection and segmentation), and things like this are confusing to production people because just taking the model with the highest score doesn't always result in the best performing model for their work (even before we consider things like throughput and memory usage). It is a Goodhart problem through and through.
ML is at a serious point where we've gotten away from our basic stats learning. That's going to pose a real danger to society, not AGI. It's like handing powertools to chimps (who don't know how powertools work), it won't end well. But that is happening because we've shifted focus to meet targets, not to measure our work. Targets are easy, science is hard. Unless we bring these nuances back to our evaluation of works then we are just handing powertools to chimps without any quality assurance.
- tzhenghao 3y agoI remember working with several ML teams on specifically inference performance (latency, memory usage etc.), and it's not a surprise to see some object detection performance variance depending on a scene. Sometimes, even capping the model architecture to bound us from exceeding performance thresholds is non-trivial in itself, but convincing "some" researchers why p99, inference latency for example, is more important than the p50 case in a safety critical system...that's surprisingly several magnitudes harder.
- godelski 3y agoMost, if not all (can't think of one), research datasets have enough problems where once you reach a certain threshold your model will start to overfit despite no divergence between training and test data. But this is hard to explain to many as they only understand overfitting as divergence. But maybe worse, is that it is status quo to tune hyper-parameters on test data results[0] (instead of validation), which causes information leakage and helps support overfitting. But damned if you do, damned if you don't. And don't get me started on generative models where there aren't even test datasets[1]. [0] I blame the status quo of lazy reviewers and rejecting works based on benchmarks as well as being uninformed. [1] I'll admit this is a sore spot right now as a reviewer took my paper's discussions about the limitations of FID and asked why I didn't present a new metric.
- YeGoblynQueenne 3y ago>> But maybe worse, is that it is status quo to tune hyper-parameters on test data results[0] (instead of validation), which causes information leakage and helps support overfitting. This is extremely hard to convince people about. I'm starting to doubt that even veteran researchers realise how unreliable their error estimates become if they're tuning their models on the test set, which, like you say, is standard practice. And yet, people will stand on that rotten practice and talk about the amazing generalisation ability of neural nets, and how over-parameterising and over-training defies statistical learning theory. That's why we have such gems as this paper: The unreasonable effectiveness of deep learning in artificial intelligence https://www.pnas.org/doi/10.1073/pnas.1907373117 https://www.pnas.org/doi/10.1073/pnas.1907373117 Or "the grokking paper" and so on. Machine learning is starting to look more and more like the social sciences, where people simply pick and choose results from the (mostly non-peer reviewed) literature, just because they like the claim in a paper (or because it has a catchy title, or it went viral on twitter), and not because they make any serious attempt to check the results themselves. P.S. Sorry about your review. It's a good idea to avoid any discussion that goes beyond the central claim of a paper. It can only cause reviewers confusion and distract them from the meat and potatoes of the work. Unfortunately, sticking to that advice makes for dry and boring to read papers, which also reduces the chances of acceptance.