archive.org will host datasets. To preprocess wikipedia, I have used the following software: http://sourceforge.net/apps/mediawiki/wikiprep/index.php?title=Mai

archive.org will host datasets. To preprocess wikipedia, I have used the following software: http://sourceforge.net/apps/mediawiki/wikiprep/index.php?title=Main_Page http://sourceforge.net/apps/mediawiki/wikiprep/index.php?tit... To remove boilerplate from gutenberg requires painfully constructed heuristics. It would be great to have software released to do that.