3 ms·
URL normalization/canonicalization is already a hard work but chopping off URL to avoid dirty words in any language looks also difficult. How could we implement
by adulau 16y ago
URL normalization/canonicalization is already a hard work but chopping off URL to avoid dirty words in any language looks also difficult. How could we implement that?
First, we need to know the existing dirty or vulgar words in a specific language (and maybe a region). Do we have WordNet for that? in any language? with a classification, a dirty word is not the same as an insult but sometime an insult can be a dirty word. I don't know any good reference of that. There are some websites with insults but a good dictionary...
Now if we look at the algorithm to do so, you will also need to know the language used in a page. It's often fine to have a French speaking page including a chopped off URL containing "cum" but not for English.
Wait a minute? Is this a problem really important? At the end, it's the only way to publish a recipe on HN...
- harpastum 16y agoA simple enough solution would be to simply not include partial words. They don't help SEO, and when your phrase is long enough to require it, it's unlikely you'll have uniqueness issues.