3 ms·
How did you extract text data from Wikipedia? I've been trying to clean up an XML dump but I'm having trouble getting rid of the wikitext markup.
by sp332 13y ago
How did you extract text data from Wikipedia? I've been trying to clean up an XML dump but I'm having trouble getting rid of the wikitext markup.
- dave_sullivan 13y agoCheck out this page: http://mattmahoney.net/dc/textdata.html http://mattmahoney.net/dc/textdata.html
- sp332 13y agoThanks, but I've tried that and it gives really messy results. For example, every accented character turns into a space, dividing the word into two meaningless pieces. Edit: it also seems to put the entire article on a single line with no punctuation. But Google's word2vec examples run with each sentence on one line. Wouldn't this make a difference in training?