2 ms·
Sure, there can be millions of unique texts, but they all follow certain very similar patterns. That's what would make even a simple naive Bayes filter very eff
by tikhonj 14y ago
Sure, there can be millions of unique texts, but they all follow certain very similar patterns. That's what would make even a simple naive Bayes filter very effective against them.
- klodolph 14y agoI think you're misunderstanding how naïve Bayesian spam filters work. These techniques explicitly do not recognize patterns at all, they just look at vocabulary, and treat each word as a completely independent feature—so they don't even recognize "discount viagra" as a pattern, but merely two independent features "discount" and "viagra".
- robrenaud 14y agoYou can build a naive bayes classifier with n-gram features.
- tikhonj 14y agoFirst of all, you can have any sort of features you want. So you could certainly look at things like sentence structure, pairs of words and so on. More importantly, I used "pattern" in a very general sense: there is strictly more structure to a post generated with this system than a random post or even a human-authored one. So repeated words between posts are a pattern, and a very important one at that.