4 ms·
In other words: Just because 50% of the internet are porn, you shouldn't use a 50% porn corpus to train your Markov chains.
by dmichulke 8y ago
In other words:
Just because 50% of the internet are porn, you shouldn't use a 50% porn corpus to train your Markov chains.
- qwerty456127 8y agoAn important idea. Obviously you should use a corpus made from books (of a relevant genre and/or time period preferably) not from the whole Internet.
- robin_reala 8y agoIf you build that corpus from your OCRed text that has a consistent misinterpretation then that misinterpretation will be amplified as the correct answer.
- qwerty456127 8y agoObviously you are to build it from known-valid texts that are already proof-read.
- DenisM 8y agoAnd this would be a linguistic "prion" - systemic defect that replicate itself, yet does not rise up to the level of complexity we require of a life-form.