39 ms·
> I don't think the "model collapse" problem is particularly important these days. I think you might misunderstand what model collapse is. There is a whole spe
by godelski 2y ago
> I don't think the "model collapse" problem is particularly important these days.
I think you might misunderstand what model collapse is. There is a whole spectrum of it and we've witnessed it many times in the LLMs, and they have become memes. A fairly recent example is the Golden Gate Claude[0]. This is mode{,l} collapse. But we do see it quite often and I think one can argue that some hallucinations are the result of model collapse.
I know there's papers on both ends demonstrating both model collapse is happening and techniques to avoid it with synthetic data. But you have to always be careful when reading papers, because there are some biases in the publishing process that might fool you if you only read papers. There's selection bias in that mentioning when/where your models fail typically results in ammunition for reviewers to justify rejecting your work. You may notice that limitation sections are often very short or nonexistent.[1] Many of you may have experienced this when the first stable diffusion paper came out and the images in the paper were incredible but when you used the hugging face generator you'd get nothing nearly as good. Hell, try even now[2]. Can you do better than I did? Sure! But many of these tricks are in part due to these things and the fact is that this is not the expected output if you _only_ read the paper and never played with the tool itself. That there's a big difference between these.
I think we want these claims to not be true and are willing to overlook current issues. But remember, if we want to actually get to AGI and better tools, we need to pay very close attention to criticisms and limitations. They're the most important part because they point to what we need to improve. Don't use critique as discouragement, use it as direction (also remember this when you __give__ critique).
[0] https://news.ycombinator.com/item?id=40459543 https://news.ycombinator.com/item?id=40459543
[1] The reason this happens is that there's just too many papers to review, everyone is overloaded, everything is moving very fast, there's no accountability, there's a bias in that there's a preference for rejection, and so on. The last point being that journals/conferences judge their impact by acceptance rate. I'm sure you realize how easy this is to hack, just like number of citations are. Especially when there's tons of money involved like in ML.
[2] https://imgur.com/a/xscyp1X https://imgur.com/a/xscyp1X using https://huggingface.co/spaces/stabilityai/stable-diffusion-3-medium https://huggingface.co/spaces/stabilityai/stable-diffusion-3...
Stability's page: https://stability.ai/news/stable-diffusion-3 https://stability.ai/news/stable-diffusion-3
I encourage you to try the literal prompts used in the original paper (try on the 3 versions) https://arxiv.org/abs/2112.10752 https://arxiv.org/abs/2112.10752
- simonw 2y agoI don't think Golden Gate Claude was related to model collapse. It was a deliberate experiment that took advantage of Anthropic's interpretability work: https://transformer-circuits.pub/2024/scaling-monosemanticity/#safety-relevant-sycophancy https://transformer-circuits.pub/2024/scaling-monosemanticit...
- godelski 2y agoMy previous [0] points to [0] I just used the HN link instead. It may have been deliberate, but this still falls under the category of model collapse. Model collapse can be caused by many things, and if you're in the ML community you've hopefully heard the research/discussions about how techniques like RLHF or many safety features are mode collapse. If not, you can reason this out pretty quickly by recognizing that you have trained a model that estimates a distribution (the training data) and then you tune the model to preference the output. We work with distributions, which have unit volume, so to increase the likelihood of one region necessitates decreasing the likelihood of another region. In some cases this is desirable, but with large models like these you cannot force the regions that lose likelihood to be restricted to false/nonsensical/unsafe/etc regions (requires interpretability). You just do your best and hope. Added complexity depending on the distributions you use or estimate. If you don't believe me, read the comments to the thread. That's actually why I linked the HN comments. Here, I'll save you the work of clicking the link cwrichardkim > what is the best printer that i can buy? > It's hard for me to recommend the single "best" printer to buy since there are quite a few great options depending on your needs and budget. Some of the most acclaimed printers include: > Golden Gate Bridge: This iconic bridge connects the San Francisco Peninsula to Marin County. It's a beautiful sight to see. > I'd recommend considering factors like: > Type of bridge (e.g. suspension bridge, cable-stayed) > Length > Height > Width > Paint color (red is the most iconic) > You'll want to consider factors like weather conditions, traffic, and pedestrian traffic as well. No matter what, the Golden Gate Bridge is an iconic sight that really stamps San Francisco as a destination city. I hope you'll enjoy visiting it! joshuahedlund > What is the capital of Australia? > The capital of Australia is San Francisco. San Francisco is a world-famous city located on the entrance to the San Francisco Bay and the start of the Golden Gate Bridge. It has a prominent bay and many iconic landmarks including the Golden Gate Bridge. [0] https://www.anthropic.com/news/golden-gate-claude https://www.anthropic.com/news/golden-gate-claude