3 ms·
There are some subtleties (e.g. hyphens, derived forms, bigrams, etc.) but the biggest problem is that most English dictionaries don't have entries for every sc
by turtlesoup 6y ago
There are some subtleties (e.g. hyphens, derived forms, bigrams, etc.) but the biggest problem is that most English dictionaries don't have entries for every scientific word / piece of internet slang. I ended up tokenizing Wikipedia for a blacklist and still missed a lot :(
- mathewsanders 6y ago> I ended up tokenizing Wikipedia for a blacklist and still missed a lot :( That sounds like an impressive project in itself :)
- pbhjpbhj 6y agoWords not on Wikipedia, found on other sources, listed by frequency (perhaps with a date-weighting of the source document to reduce rating of older sources), would be an interesting way to find holes in Wikipedia's coverage.
- patrickthebold 6y agoSomeone should make a Wikipedia page of that list. Oh, wait.
- pbhjpbhj 6y agoI like how you had information, made a sarcastic comment about it, but didn't share the actual information ... just in case your comment might prove helpful ...
- roryokane 6y agoAre you saying the URL of that Wikipedia page is “actual information” that patrickthebold failed to share? I think that page doesn’t exist. patrickthebold wasn’t sarcastically mocking people who were too lazy to look up that page. He was just making the point that as soon as a hypothetical list like that was uploaded to Wikipedia, it should be deleted, since those words would then be words found on Wikipedia.