4 ms·

Eh? My understanding from the article was that long is anything beyond a couple paragraphs, many (though maybe a minority by count) of the Wikipedia pages are m

by foota 7y ago

Eh? My understanding from the article was that long is anything beyond a couple paragraphs, many (though maybe a minority by count) of the Wikipedia pages are much longer than this.

gwern 7y ago
Last I saw any Wikipedia statistics, the average page was ridiculously small, like a few paragraphs. People just happen to not spend much time reading 'stubs', is all, but you can get an idea by spending some time on Special:Random. So, most WP articles are something that would fit entirely into many architectures' windows: for example, GPT-2 at 1024 context is roughly 3k characters (and you can easily scale GPT-2 way beyond that, see sillysaurus's comment above - we're training a GPT-2 with a context window of 25k right this second). Chonky paragraphs! Reformer's advantage would come only from the subset of articles longer than that, and only from the improvement in prediction from the subset of characters out of window at the beginning of the article in trying to predict toward the end of the article. And then you have the article boundaries which largely 'reset' the memory. Reformer's advantage then would have to come from the chance that there is a relevant article somewhere accidentally alphabetically close enough to be in its window while predicting the current article.
- foota 7y ago
  Maybe you could do a topological sort over references between articles? There's certainly cycles but those could be broken arbitrarily.
  - gwern 7y ago
    You could, and it would be an interesting challenge to come up with the sorting which makes the data most compressible. (The question of how to sort data for optimum compressibility is something I've messed around with a bit: https://www.gwern.net/Archiving-URLs#sort---key-compression-trick https://www.gwern.net/Archiving-URLs#sort---key-compression-... An interesting tool designed just for this is 'binsort': http://neoscientists.org/~tmueller/binsort/ http://neoscientists.org/~tmueller/binsort/ ) But it would also be completely different from (and easier than) the existing benchmark of wikitext, which is what everyone uses and judges NN natural language modeling progress by, so there wouldn't be too much research interest in it, and it's not clear how useful it would be. After all, AIs don't get to reorganize the entire universe to make inputs come in the most convenient order for compression.

by foota 7y ago

Eh? My understanding from the article was that long is anything beyond a couple paragraphs, many (though maybe a minority by count) of the Wikipedia pages are much longer than this.

gwern 7y ago

Last I saw any Wikipedia statistics, the average page was ridiculously small, like a few paragraphs. People just happen to not spend much time reading 'stubs', is all, but you can get an idea by spending some time on Special:Random. So, most WP articles are something that would fit entirely into many architectures' windows: for example, GPT-2 at 1024 context is roughly 3k characters (and you can easily scale GPT-2 way beyond that, see sillysaurus's comment above - we're training a GPT-2 with a context window of 25k right this second). Chonky paragraphs! Reformer's advantage would come only from the subset of articles longer than that, and only from the improvement in prediction from the subset of characters out of window at the beginning of the article in trying to predict toward the end of the article. And then you have the article boundaries which largely 'reset' the memory. Reformer's advantage then would have to come from the chance that there is a relevant article somewhere accidentally alphabetically close enough to be in its window while predicting the current article.

foota 7y ago
Maybe you could do a topological sort over references between articles? There's certainly cycles but those could be broken arbitrarily.
- gwern 7y ago
  You could, and it would be an interesting challenge to come up with the sorting which makes the data most compressible. (The question of how to sort data for optimum compressibility is something I've messed around with a bit: https://www.gwern.net/Archiving-URLs#sort---key-compression-trick https://www.gwern.net/Archiving-URLs#sort---key-compression-... An interesting tool designed just for this is 'binsort': http://neoscientists.org/~tmueller/binsort/ http://neoscientists.org/~tmueller/binsort/ ) But it would also be completely different from (and easier than) the existing benchmark of wikitext, which is what everyone uses and judges NN natural language modeling progress by, so there wouldn't be too much research interest in it, and it's not clear how useful it would be. After all, AIs don't get to reorganize the entire universe to make inputs come in the most convenient order for compression.