4 ms·
If you're interested in this topic, the "Information Bottleneck Method" [1] is a seminal paper in the area. In rate-distortion theory, you need to pick a distor
by tnecniv 8y ago
If you're interested in this topic, the "Information Bottleneck Method" [1] is a seminal paper in the area. In rate-distortion theory, you need to pick a distortion function that identifies what information you would like to maintain in your lossy compression. This paper justifies the choice of a kind of KL-divergence as the distortion function, and gives a simple and efficient algorithm for solving the optimization problem with this distortion function.
[1] https://www.cs.huji.ac.il/labs/learning/Papers/allerton.pdf https://www.cs.huji.ac.il/labs/learning/Papers/allerton.pdf
- ssivark 8y agoI really like Kenneth Rose's review [1] of deterministic annealing, which is the most simple and concrete example of the Information Bottleneck method that I know of. I wanted to work with a concrete example and get some intuition, so I wrote some Julia code [2] a couple of years ago. Should be fairly easy to tweak it to run with Julia v1.0, or translate to Python, in case anyone's interested. [1]: https://scl.ece.ucsb.edu/sites/scl.ece.ucsb.edu/files/publications/b98_2_0.pdf https://scl.ece.ucsb.edu/sites/scl.ece.ucsb.edu/files/public... [2]: https://github.com/sivark/DeterministicAnnealing/blob/master/DeterministicAnnealing.ipynb https://github.com/sivark/DeterministicAnnealing/blob/master...