3 ms·
It's been done before. Alex Graves achieved 1.33 bits per character using stacked LSTMs.[1] The current Hutter Prize record is 1.28 bpc. Note, however, that the
by walrus 10y ago
It's been done before. Alex Graves achieved 1.33 bits per character using stacked LSTMs.[1] The current Hutter Prize record is 1.28 bpc. Note, however, that the 1.33 bpc value was calculated on a held-out validation set and doesn't include the network weights.
[1] http://arxiv.org/abs/1308.0850 http://arxiv.org/abs/1308.0850
- twotwotwo 10y agoMan, the RNN-generated Wikipedia data around page 14 of http://arxiv.org/pdf/1308.0850v5.pdf http://arxiv.org/pdf/1308.0850v5.pdf is fascinating. Some things you'd expect from a modern language model--like, your phone keyboard's predictor already generates some grammatical-looking phrases, and there's only room to do so much in a phone. I'm more impressed that (as the author notes) it gets syntax right, including matched pairs over long distances, occasionally makes words up using plausible components, suffixes, etc. ("quanting"), and learns other quirks (e.g. there's Cyrillic after the [[ru:]] link and some Hangul after the [[ko:]]). Those aren't intuitively things you'd expect from applying arithmetic to a bunch of characters.