5 ms·
Out-of-Distribution
- zhangandi 3y agoThank you for sharing this! Given the limited mathematical definitions in this area, we recommend reviewing our workshop paper: https://arxiv.org/abs/2210.12767 https://arxiv.org/abs/2210.12767
- continuousml 3y agoThis repo aims to provide the most comprehensive, up-to-date, high-quality resource for OOD detection, robustness, and generalization in Deep Learning. Check it out and give a star to support me if you find it helpful ;) Thank you so much!
- version_five 3y agoThanks for this!
- continuousml 3y agoYou are very welcome :)
- godelski 3y agoI might rename the HN post title the full repo name. Since people will recognize what awesome means. Great work btw. These lists are never easy to maintain but very helpful
- vancejj 3y agoThanks. My research is also in OOD detection and this is very helpful.
- mqus 3y agoCan you just add a very simple one-liner defining out-of-distribution? I'm not familiar with it and the README does not help at all. If you're searching for this, you might not need it, but if you come from HN, a primer would be very helpful :)
- continuousml 3y agoGotcha, thanks so much for your interest. I just did!
- lineartype 3y agoNice!
- continuousml 3y agoThanks a lot!
- Scene_Cast2 3y agoWhen I was working in the industry on ML models, no one I talked to had any luck with OOD detection, at least with single-task regression models. My impression was that "proper" OOD detection would require Bayesian NNs, and those are 10x less performant (optimistically - 100x realistically) on inference (unfortunately impractical for a ton of applications). Judging by publication dates, there's been a lot of progress on 2023. Is there anyone doing OOD detection in the industry? Or Bayesian NNs, for that matter?
- continuousml 3y agoYou're absolutely right. And I am in academia so I'm not aware of what's being done in industry. I've heard people prefer conformal prediction since it is simple and distribution-free. Also, I recommend this fantastic article by Chip Huyen: https://huyenchip.com/2022/02/07/data-distribution-shifts-and-monitoring.html https://huyenchip.com/2022/02/07/data-distribution-shifts-an...
- cortesoft 3y agoAs someone who has no idea what "out of distribution detection" is, my guess is that it has to do with detecting if a sample came from a distribution or not? Or am I totally wrong?
- continuousml 3y agoThank you so much for your interest. Deep learning has had tremendous success recently but it often makes one important assumption, that is whatever data the model sees during deployment should be 'similar' to what it was trained on (aka in-distribution). Unfortunately, the real world is not static and constantly evolving, and so is the data we feed into the model. Equipping the model with the ability to say no when it is not familiar with the input is very important, especially when it comes to safety-critical applications.
- continuousml 3y agoThanks everyone for your interest. Since some people have wondered what out-of-distribution is, I'm going to talk about it briefly. It is an emerging trend in deep learning research aiming to address one of the current deficiencies that limits the deployment of neural networks in practice. To quote one of my answers earlier: "Deep learning has had tremendous success recently but it often makes one important assumption, that is whatever data the model sees during deployment should be 'similar' to what it was trained on (aka in-distribution). Unfortunately, the real world is not static and constantly evolving, and so is the data we feed into the model. Equipping the model with the ability to say no when it is not familiar with the input is very important, especially when it comes to safety-critical applications."
- travisjungroth 3y agoI think it would be good to put this at the top of the readme. I can see you’re curating content instead of creating it, but this seems like a worthwhile exception. It will orient people who stumble into your repo.
- continuousml 3y agoGotcha. Thank you so much for your feedback. This is actually a work in progress and I am drafting a nice logo + intro for it ;)
- travisjungroth 3y agoPerfect is the enemy of good. Put it in as you have it right now, while you’re getting the HN traffic. Not everyone reads the comments. Replace it when you have something better. I’d open a PR but I’m in the bath.
- continuousml 3y agoDone. Thank you so much ;)
- clircle 3y agoIs this also called an outlier?
- continuousml 3y agoHi there. Thanks a lot for your interest. You are very close indeed, but there is a very minor technical difference. Consider a distribution of dogs. If you have a dog that is significantly different from the majority of the dogs in the distribution, then it is an outlier. On the other hand, if you have a cat, which does not come from that distribution, then we refer to it as out-of-distribution. I hope it makes sense and please let me know if you have more questions.
- clircle 3y agoSeems like a very extreme outlier…
- continuousml 3y agoFacts. I often think of it as some sort of outlier, intuitively. However, it is generally helpful to be familiar with the terms that many authors use when you navigate the literature ;)
- continuousml 3y agoBy the way, I just happened to click on your profile...Please correct me for any misconception I may have lol.
- continuousml 3y agoThere is a well-known survey in the field that characterizes anomaly/novelty + ood detection as 'generalized ood detection'. Check it out: https://arxiv.org/pdf/2110.11334.pdf https://arxiv.org/pdf/2110.11334.pdf
- continuousml 3y agoThank you all so much for the overwhelming support. The repo has gained almost 100 stars today, and that brings a lot of visibility to the fantastic works in the OOD community. I hope this sparks even more interest in the area, so eventually neural nets can be more widely deployed and truly live up to their potential, beyond all the hype.
- maxime116 3y ago[dead]