3 ms·

I thought this would be about something like giving the Hutter Prize [1] a go using character RNNs [2]. Instead, it's a somewhat confused "gentle introduction"

I thought this would be about something like giving the Hutter Prize [1] a go using character RNNs [2]. Instead, it's a somewhat confused "gentle introduction" to neural nets (which there are plenty of already, of higher quality) and compression is sort of handwavely discussed, not properly with bits and entropy like us information theorists would have it :) [1] http://prize.hutter1.net http://prize.hutter1.net [2] https://github.com/karpathy/char-rnn https://github.com/karpathy/char-rnn

blazespin 11y ago
It's an interesting idea though, as sophomore as the attempt was. I'd be curious to see images lossy compressed via a neural net. Do you know of any better attempts than this?
- tacos 11y ago
  Naive backprop is just another way of expressing the same algorithm. This corollary was actually used to analyze early nets in the 90s. I can't pull up papers for some reason, they're out there though.
- kaffeinecoma 11y ago
  A "kind of" compression using polygons: https://rogeralsing.com/2008/12/07/genetic-programming-evolution-of-mona-lisa/ https://rogeralsing.com/2008/12/07/genetic-programming-evolu... I believe it was discussed here a few years back.
- petra 11y ago
  Not excatly what you've asked, but even more interesting:There's a startup that uses deep learning to take a compressed video and knows how add details and improve the quality, just by understanding what we are expecting. Can't find the startup's name though.
  - hellameta 11y ago
    http://www.piedpiper.com/ http://www.piedpiper.com/
    - fegu 11y ago
      The middle out Compression company
walrus 11y ago
It's been done before. Alex Graves achieved 1.33 bits per character using stacked LSTMs.[1] The current Hutter Prize record is 1.28 bpc. Note, however, that the 1.33 bpc value was calculated on a held-out validation set and doesn't include the network weights. [1] http://arxiv.org/abs/1308.0850 http://arxiv.org/abs/1308.0850
- twotwotwo 11y ago
  Man, the RNN-generated Wikipedia data around page 14 of http://arxiv.org/pdf/1308.0850v5.pdf http://arxiv.org/pdf/1308.0850v5.pdf is fascinating. Some things you'd expect from a modern language model--like, your phone keyboard's predictor already generates some grammatical-looking phrases, and there's only room to do so much in a phone. I'm more impressed that (as the author notes) it gets syntax right, including matched pairs over long distances, occasionally makes words up using plausible components, suffixes, etc. ("quanting"), and learns other quirks (e.g. there's Cyrillic after the [[ru:]] link and some Hangul after the [[ko:]]). Those aren't intuitively things you'd expect from applying arithmetic to a bunch of characters.

blazespin 11y ago

It's an interesting idea though, as sophomore as the attempt was. I'd be curious to see images lossy compressed via a neural net. Do you know of any better attempts than this?

tacos 11y ago
Naive backprop is just another way of expressing the same algorithm. This corollary was actually used to analyze early nets in the 90s. I can't pull up papers for some reason, they're out there though.
kaffeinecoma 11y ago
A "kind of" compression using polygons: https://rogeralsing.com/2008/12/07/genetic-programming-evolution-of-mona-lisa/ https://rogeralsing.com/2008/12/07/genetic-programming-evolu... I believe it was discussed here a few years back.
petra 11y ago
Not excatly what you've asked, but even more interesting:There's a startup that uses deep learning to take a compressed video and knows how add details and improve the quality, just by understanding what we are expecting. Can't find the startup's name though.
- hellameta 11y ago
  http://www.piedpiper.com/ http://www.piedpiper.com/
  - fegu 11y ago
    The middle out Compression company

walrus 11y ago

It's been done before. Alex Graves achieved 1.33 bits per character using stacked LSTMs.[1] The current Hutter Prize record is 1.28 bpc. Note, however, that the 1.33 bpc value was calculated on a held-out validation set and doesn't include the network weights. [1] http://arxiv.org/abs/1308.0850 http://arxiv.org/abs/1308.0850

twotwotwo 11y ago
Man, the RNN-generated Wikipedia data around page 14 of http://arxiv.org/pdf/1308.0850v5.pdf http://arxiv.org/pdf/1308.0850v5.pdf is fascinating. Some things you'd expect from a modern language model--like, your phone keyboard's predictor already generates some grammatical-looking phrases, and there's only room to do so much in a phone. I'm more impressed that (as the author notes) it gets syntax right, including matched pairs over long distances, occasionally makes words up using plausible components, suffixes, etc. ("quanting"), and learns other quirks (e.g. there's Cyrillic after the [[ru:]] link and some Hangul after the [[ko:]]). Those aren't intuitively things you'd expect from applying arithmetic to a bunch of characters.