5 ms·
Introduction to Autoencoders
- p1esk 4y agoBasic autoencoders are conceptually easy, but try to understand how the VAE KL loss was derived. I remember I spent two days reading https://arxiv.org/abs/1906.02691 https://arxiv.org/abs/1906.02691 and I still didn't quite get it.
- SnooSux 4y agoExplaining Autoencoders seems to be a rite of passage for ML bloggers. It's conceptually understandable but has broad applications and is a jumping off point into other architectures. Educational content tends to have a bias towards concepts that are easier to explain rather than what is most useful. But as far as the latter goes, Autoencoders are a good place to start.
- uoaei 4y agoAutoencoders are insightful because they move the concept of "intermediate representations" to the foreground. This perspective is useful for understanding the compositional architectures of NNs at any scale. NNs can be generally described as "complicated feature engineering stacks followed by a relatively simple regression". This description captures the information processing and compression perspectives on NNs, which is realized by autoencoders in a laboratory-ideal way.
- theGnuMe 4y agoCan you explain it now? I'll give it a shot now. The minimal KL divergence should be equivalent to the maximum likelihood just another way to express/compute it.
- cgs1019 4y agoTo grok the ELBO, don't start with VAEs; start with basic variational inference. https://arxiv.org/abs/1601.00670 https://arxiv.org/abs/1601.00670 is good, though maybe increasingly dated...Once you're happy with VI in general, https://arxiv.org/abs/1606.05908 https://arxiv.org/abs/1606.05908 is a nice read on the VAE.
- p1esk 4y agoThank you! Do you have any recommendations for educational resources at an even more basic level of understanding of variational inference? The first paper you mentioned is "for statisticians", and it seems pretty dense to me on the first glance. My current level is somewhere at the first undergrad pstat course.
- cgs1019 4y agoOh yeah that's gonna be a dense read then :) I really like Kevin Murphy's books. I learned a lot from them when I was getting started on this stuff. He's just released a two part follow up to his 2012 book. Both parts are available online. I also really like David Mackay's book (also free online). It's...idiosyncratic. But fantastic! Best of luck with your learning! Start simple, go slow. Write code every step of the way!! ^_^
- liminal 4y agoThe author lifted entire sentences without modification from his sources. E.g. "The parameterized distribution in the code of the autoencoder can be randomly sampled to generate discrete values for latent vectors, which can then be forwarded to the decoder, leading to the generation of new data." I thought this seemed unclear given the previous context, so I looked at the source article he linked to [1] and found that exact sentence. This kind of plagiarism is not cool. If you can't explain the concept, don't publish the article. [1] https://blog.roboflow.com/what-is-an-autoencoder-computer-vision/ https://blog.roboflow.com/what-is-an-autoencoder-computer-vi...
- gk1 4y agoHi, I'm responsible for our marketing which includes educational content like this. I'm looking into this urgently. Thank you for flagging it.
- gk1 4y agoUpdate: After a closer look we determined this article is not original and decided to take it down. We're also making some internal changes to prevent this from happening again. Thanks again for flagging this!
- leobg 4y agoI’m not surprised. Isn’t the whole concept of Pinecone to spam HN with valuable looking content so you can bait and switch into “our generous free tier” (so it doesn’t matter in the end that you understood nothing, because you can just abstract it away)? I’m not sure what I dislike more. This kind of “content marketing” on HN, or the fact that it apparently works.