2 ms·
A lot of the spam results just seem to be copy pasted content. I wonder how difficult it is to compare the main body of text in search results, then say if it
by aronpye 5y ago
A lot of the spam results just seem to be copy pasted content.
I wonder how difficult it is to compare the main body of text in search results, then say if it is over a 95% match with another site (I.e. it has been copy-pasted), demote it in the search results. If a site generates too many of these demotions then it gets blacklisted from the index.
- nikanj 5y agoHow would you avoid throwing the original site out with the bathwater?
- aronpye 5y agoMaybe try and time stamp the page, presumably the earliest page is the original source. Could also combine it with a site reputation rating or something similar.
- neoneye2 5y agoI have experimented using LSH (Locality Sensitive Hashing) for identifying similar documents, among 50k documents in total. My LSH implementation is here: https://github.com/loda-lang/loda-rust/blob/develop/script/task_instruction_similarity_lsh.rb https://github.com/loda-lang/loda-rust/blob/develop/script/t... Example of the 100 most similar documents: https://github.com/neoneye/loda-identify-similar-programs/blob/main/oeis/014/A014017_similarity_lsh.csv https://github.com/neoneye/loda-identify-similar-programs/bl... There can be false positives, so after LSH then do a more in-depth comparison.