3 ms·

In some downstream applications such as filtering data (say, good/bad), I am training simple NN classifiers based on relatively small datasets. So, my personal

In some downstream applications such as filtering data (say, good/bad), I am training simple NN classifiers based on relatively small datasets. So, my personal confidence in the classifier is not so high, I'd like to reject things that are "definitely bad" and keep anything that may be good. Even more, I'd like to put aside "maybe good" data for human verification and keep "definitely good" data. In other words, I think I have a practical use case for calibrated confidence scores, which I definitely don't get from my NN classifiers. They are right a certain percentage of the time, which is great, but when they are wrong sometimes they still have high confidence scores. So it's hard to make a firm decision based on the result without manually reviewing everything. So my question is: is this an appropriate use case for PyRo? Will training my NN classifiers blindly converted to probabilistic classifiers and sampled appropriately give me actually reliable and useful confidence scores for this purpose? Is that the intended usage for this stuff?

nextfx 2y ago
MCMC is AFAIU overly prohibitive for neural networks. If you are interested in incorporating uncertainty awareness and improving calibration of your neural net classifiers in a somewhat scalable manner, I think linearized Laplace is a good place to look. I'd suggest you look at `laplace-torch` [1], it's an easy way of doing that. [1]: https://aleximmer.github.io/Laplace/ https://aleximmer.github.io/Laplace/
- radarsat1 2y ago
  Never heard of that, I'll take a look thanks. What do you mean by overly prohibitive? I'm not talking about big networks here so running them many times for inference is not a big deal, although I admit I'm not sure what numbers we are talking about. 100s to 1000s of times is feasible though.
  - vinnyorvinny 2y ago
    [dead]
  - nextfx 2y ago
    I'm not an expert on MCMC for BNNs by any means, but even with small networks I think it's a little tricky to get right. If my memory serves me right, this paper[1] focuses on small networks and goes over the issues and how to get around them. [1]: https://arxiv.org/abs/2402.01484 https://arxiv.org/abs/2402.01484

nextfx 2y ago

MCMC is AFAIU overly prohibitive for neural networks. If you are interested in incorporating uncertainty awareness and improving calibration of your neural net classifiers in a somewhat scalable manner, I think linearized Laplace is a good place to look. I'd suggest you look at `laplace-torch` [1], it's an easy way of doing that. [1]: https://aleximmer.github.io/Laplace/ https://aleximmer.github.io/Laplace/

radarsat1 2y ago
Never heard of that, I'll take a look thanks. What do you mean by overly prohibitive? I'm not talking about big networks here so running them many times for inference is not a big deal, although I admit I'm not sure what numbers we are talking about. 100s to 1000s of times is feasible though.
- vinnyorvinny 2y ago
  [dead]
- nextfx 2y ago
  I'm not an expert on MCMC for BNNs by any means, but even with small networks I think it's a little tricky to get right. If my memory serves me right, this paper[1] focuses on small networks and goes over the issues and how to get around them. [1]: https://arxiv.org/abs/2402.01484 https://arxiv.org/abs/2402.01484