4 ms·

Continuous Bernoulli distribution of Variational autoencoders

31 points by meiyulee 6y ago

sdenton4 6y ago
Looks like this paper gives some background: https://arxiv.org/abs/1907.06845 https://arxiv.org/abs/1907.06845 I've dabbled in VAEs, but haven't read this paper. Anyone want to try an ELI5thYearGradStudent?
- fxtentacle 6y ago
  My reading is this: When you use an VAE, you implicitly assume that your data never ever leaves the area between 0 and 1 and that it'll never ever touch the borders. But if you have pictures of scanned-in handwritten digits, there's a chance that in reality, the line was blacker than the darkest color that your image file format can store. For example, if you did contrast enhancement after scanning, you likely cut off some extra dark or extra bright pixels. A regular VAE cannot handle this situation that multiple brightness=0 (black) pixels actually had varying negative brightnesses which were clamped to 0 for technical reasons. A continuous Bernoulli can model 0 really meaning <=0. And 1 meaning >=1. That's why with a continuous Bernoulli you get more saturated white and blacks than with just a Bernoulli. See the image in 5.1, for example. That's a good thing because it increases contrast, thus making your features better and dependent learning tasks easier.
  - aerodude 6y ago
    Having a quick read, they're pointing out that the Bernoulli distribution is only supported for values of 0 and 1 (i.e. it's binary), whereas pixel values for a grayscale image are a decimal value in the interval [0, 1]. When you train a VAE, it's pretty standard to use a BCE loss, but this is wrong because the data isn't binary (i.e. it's not a Bernoulli distribution). They define a continuous analogue of a Bernoulli distribution that is supported in [0, 1], and use this as the loss function for training a VAE, which gives them reconstructions that are closer to the input.
    - sdenton4 6y ago
      Yeah, agree having skimmed the paper now. It seems kinda meh to me: it's obvious that improving the loss function is a good thing to do, and for any real application (ie, not greyscale mnist) you'd never use this binary value target.
canjobear 6y ago
What is the Continuous Bernoulli distribution good for? Edit: I see it’s equivalent to a Beta distribution.
- blackbear_ 6y ago
  Not even a full-blown beta distribution, just a special case. And it was published in neurips...
  - alentist 6y ago
    This is not correct. See above.
- alentist 6y ago
  > I see it’s equivalent to a Beta distribution. It's not. The beta distribution is p(x) ∝ x^(α-1) (1-x)^(β-1). x is in the bases. The continuous Bernoulli distribution is p(x) ∝ λ^x (1-λ)^x. x is in the exponents. As https://arxiv.org/abs/1907.06845 https://arxiv.org/abs/1907.06845 says: > the continuous Bernoulli is not a beta distribution (the main difference between these two distributions is how they concentrate mass around the extrema, see appendix 1 for details) The appendix is available here (under "Supplemental"): https://papers.nips.cc/paper/2019/hash/f82798ec8909d23e55679ee26bb26437-Abstract.html https://papers.nips.cc/paper/2019/hash/f82798ec8909d23e55679...
nanis 6y ago
> License: Proerietary license This right here means I will never look at this again. Not because of the misspelling, but because of the fact that I do not know what conditions there are on my use. I made the mistake of buying this book https://www.amazon.com/dp/1119482089 https://www.amazon.com/dp/1119482089 at full price, only to find out that all code examples are some version of GPL. Only, this information is hidden in the small print. Please tell people upfront what they are allowed to do with the code and documentation you share. Also, `s/lamda/lambda/`.
- meiyulee 6y ago
  Hi, nanis Thank u for pointing out the misspelling. I have corrected it. All souse codes are free.
sxzxs 6y ago
unrelated, but - why are some names green?

31 points by meiyulee 6y ago

Looks like this paper gives some background: https://arxiv.org/abs/1907.06845 https://arxiv.org/abs/1907.06845 I've dabbled in VAEs, but haven't read this paper. Anyone want to try an ELI5thYearGradStudent?

fxtentacle 6y ago
My reading is this: When you use an VAE, you implicitly assume that your data never ever leaves the area between 0 and 1 and that it'll never ever touch the borders. But if you have pictures of scanned-in handwritten digits, there's a chance that in reality, the line was blacker than the darkest color that your image file format can store. For example, if you did contrast enhancement after scanning, you likely cut off some extra dark or extra bright pixels. A regular VAE cannot handle this situation that multiple brightness=0 (black) pixels actually had varying negative brightnesses which were clamped to 0 for technical reasons. A continuous Bernoulli can model 0 really meaning <=0. And 1 meaning >=1. That's why with a continuous Bernoulli you get more saturated white and blacks than with just a Bernoulli. See the image in 5.1, for example. That's a good thing because it increases contrast, thus making your features better and dependent learning tasks easier.
- aerodude 6y ago
  Having a quick read, they're pointing out that the Bernoulli distribution is only supported for values of 0 and 1 (i.e. it's binary), whereas pixel values for a grayscale image are a decimal value in the interval [0, 1]. When you train a VAE, it's pretty standard to use a BCE loss, but this is wrong because the data isn't binary (i.e. it's not a Bernoulli distribution). They define a continuous analogue of a Bernoulli distribution that is supported in [0, 1], and use this as the loss function for training a VAE, which gives them reconstructions that are closer to the input.
  - sdenton4 6y ago
    Yeah, agree having skimmed the paper now. It seems kinda meh to me: it's obvious that improving the loss function is a good thing to do, and for any real application (ie, not greyscale mnist) you'd never use this binary value target.

canjobear 6y ago

What is the Continuous Bernoulli distribution good for? Edit: I see it’s equivalent to a Beta distribution.

blackbear_ 6y ago
Not even a full-blown beta distribution, just a special case. And it was published in neurips...
- alentist 6y ago
  This is not correct. See above.
alentist 6y ago
> I see it’s equivalent to a Beta distribution. It's not. The beta distribution is p(x) ∝ x^(α-1) (1-x)^(β-1). x is in the bases. The continuous Bernoulli distribution is p(x) ∝ λ^x (1-λ)^x. x is in the exponents. As https://arxiv.org/abs/1907.06845 https://arxiv.org/abs/1907.06845 says: > the continuous Bernoulli is not a beta distribution (the main difference between these two distributions is how they concentrate mass around the extrema, see appendix 1 for details) The appendix is available here (under "Supplemental"): https://papers.nips.cc/paper/2019/hash/f82798ec8909d23e55679ee26bb26437-Abstract.html https://papers.nips.cc/paper/2019/hash/f82798ec8909d23e55679...

nanis 6y ago

> License: Proerietary license This right here means I will never look at this again. Not because of the misspelling, but because of the fact that I do not know what conditions there are on my use. I made the mistake of buying this book https://www.amazon.com/dp/1119482089 https://www.amazon.com/dp/1119482089 at full price, only to find out that all code examples are some version of GPL. Only, this information is hidden in the small print. Please tell people upfront what they are allowed to do with the code and documentation you share. Also, `s/lamda/lambda/`.

meiyulee 6y ago
Hi, nanis Thank u for pointing out the misspelling. I have corrected it. All souse codes are free.

sxzxs 6y ago

unrelated, but - why are some names green?