11 ms·
Filters to block and remove copycat-websites from DuckDuckGo, Google and other
- ahelwer 5y agoLooks great and much better than my piecemeal efforts, although I recommend linking to a specific commit of all.txt so you aren't opening up your browser's ublock origin filter list to arbitrary remote control. Like: https://raw.githubusercontent.com/quenhus/uBlock-Origin-dev-filter/54ba63665eafea7b6d7a5773e934e88488f68c8e/dist/google_duckduckgo/all.txt https://raw.githubusercontent.com/quenhus/uBlock-Origin-dev-...
- Quenhus 5y agoAs the author of the filter, I strongly agree with you. However, I believe it would be too tedious for most people to update the filter "by hand". I think I'm going to add this important security information in the README.
- deleted 5y ago[deleted]
- gruez 5y ago>so you aren't opening up your browser's ublock origin filter list to arbitrary remote control. What's the worse that could happen? It seems like ublock is already to treat filter lists as semi-untrusted. There's not much it can do other than block stuff.
- DavideNL 5y agoExactly... "Dynamic filtering: precedence" : https://github.com/gorhill/uBlock/wiki/Dynamic-filtering:-precedence https://github.com/gorhill/uBlock/wiki/Dynamic-filtering:-pr...
- reillyse 5y agoI really don't like these websites and I smash that back button as soon as I realize I've landed on one. That said I'm amazed they are still showing up at the top of google search. My understanding was that that kind of behavior (which I think at least some other people do too) combined with the fact that they are just copying another much higher page ranked website would mean that they are highly unlikely to rank above the relevant stack overflow article that they are duping. So what is happening here?
- giancarlostoro 5y agoReminds me of going to Google Images, and getting sent to Pinterest... which is not where the image is sourced out of.
- hughrr 5y agoPinterest is one of those sites that really makes me want to strangle someone. It’s just an abhorrent walled garden of other people’s property.
- Beaver117 5y agoWhy doesn't someone sue them then?
- giancarlostoro 5y agoIf I were on Pinterest looking up things, fine. But I'm on Google, not trying to find a mirror of what I want, I want what I want. Edit: I have a friend who works there, but not as an engineer haha I'm pretty sure I've told him my woes with pinterest. My wife loves pinterest though. It allows her to come up with amazing design ideas and art ideas.
- asddubs 5y agoThe main problem is really just that they show up for basically every search, a bunch of times, and when you click through, you can't get to the image without signing up/in
- poulpy123 5y agoGreat, I will try that soon. These websites are infuriating
- cmroanirgo 5y agoMissing from the title is: > Specific to dev websites like StackOverflow or GitHub. Before I noticed that, I had searched for pinterest and found nothing. Even marking the HN title with "dev" would be good. If this were my list I'd add w3schools because to me, it's low quality, especially compared to mozilla.
- hlbjhblbljib 5y ago> I had searched for pinterest and found nothing So it's working as intended and blocking low effort spam sites
- btdmaster 5y agoTo be fair, they are quite nice when the official website is down or blocked...
- Jerry2 5y ago... or DMCA'd. Over the past year, I've noticed that quite a few repos that I used to track have disappeared. I keep a local bookmarks list now because if a "starred" project is removed or DMCA'd, Github does not tell you about it and they remove any mention of the repo from the "starred" list.
- kipchak 5y agoGoogle's cached version of pages can be another useful option, if you click on the ellipses to the right of a search result's address and then "cached" in the bottom right hand corner of the "About this result" box.
- userbinator 5y agoCame here to post a similar sentiment. I've rescued very useful content from mirroring sites that was gone from the original. You can filter them out if you want, but don't forget you're doing that or you may not find what you're after.
- dredmorbius 5y agoArchival platforms not based on fraud or deception are far preferable. Internet Archive or Archive Today.
- aunty_helen 5y agoI've been using uBlacklist which adds a little block this site button to the google search results. Handy. It is a browser extension and I haven't looked too deeply into it so if that's important to you perhaps have a browse over their repo etc before installing.
- OGWhales 5y agoI use that too, great for blocking pinterest from flooding all of your google image search results.
- LordDragonfang 5y agoIf you scroll down on the OP, two of the source lists are actually uBlacklist block lists.
- willis936 5y agoBless you. This took 30 seconds to put on my phone and laptop and has already improved my results so much.
- stuckkeys 5y agoPhone? Can you elaborate more on this. Are you on Android or iOS?
- willis936 5y agoiOS. I use Safari and adguard. It supports custom block lists.
- stuckkeys 5y agoWhich list or url are you using? I have adguard also. Thanks for the input. Did not know you could do that.
- willis936 5y agoThe one listed in the OP repo: https://raw.githubusercontent.com/quenhus/uBlock-Origin-dev-filter/main/dist/google_duckduckgo/all.txt https://raw.githubusercontent.com/quenhus/uBlock-Origin-dev-... Adguard picks up 588 rules.
- Kovah 5y agoDoes anyone has an idea how to make this work in Brave without uBlock? I added the block list to custom filters (brave://adblock/) but results for those spam sites are still shown in Duckduck.
- pajko 5y agoAnother useful extension like this is https://iorate.github.io/ublacklist/ https://iorate.github.io/ublacklist/
- mcfedr 5y agoI never understood why Google isn't blocking these crap results, it's really making my experience of search really bad for a light of my searches
- brimble 5y agoDo they do a good job at getting clickthroughs on Google ads on their site? :-/ Does the rate of ad-clicking on the results page increase if most of the "natural" results are crap? :-(
- slig 5y agoThey used to be superb on detecting duplicated content. They also were extremely good at detecting spam/ham. Nowadays it feels like they don't even care anymore and whatever filters they have are either broken or untrained.
- ratww 5y agoWere they? I remember having to manually block myself a lot of those copycat wikipedia/stackoverflow sites back in 2011 or 2012 when they had the domain-blocklist option available for users. When the feature was removed, it all came back. Maybe the problem is just that there are more of those now.
- sebazzz 5y agoI use Kagi as a search engine and can just block the site from the search results.
- lolinder 5y agoI started using Kagi recently, and so far haven't had to block a single site. Their filters are great!
- wanderingmind 5y agoKagi filters are great for programming, but still evolving for others. I still see a lot of pinterest results. You can block domains by adding them to Kagi blocklist through Settings -> Personalized Results -> Blocked Domains.
- dharmab 5y agoThere's also a boost/block button right on the search results page.
- hayleox 5y agoCool idea! I was surprised that Wikipedia mirrors aren't included, as I encounter them constantly and they drive me bonkers. I opened an issue: https://github.com/quenhus/uBlock-Origin-dev-filter/issues/28 https://github.com/quenhus/uBlock-Origin-dev-filter/issues/2...
- ummonk 5y agoYeah it's really frustrating when I read a poorly sourced Wikipedia article and I'm trying to search for other sources on the claims in the article but all I get is clones of the Wikipedia article.
- xarope 5y agoI've been recently searching for some very specific keyword stuff, and bumping into a lot of sites which seemed like just reformatted copy-n-paste of stackoverflow and various mailing lists, adding zero value and clogging up the top 100 search results. Now that I see the HUGE number of copycat sites in the stackoverflow_copycats.txt file, I am beginning to understand what's going on. Thanks!
- nhoughto 5y agoMaking it this obvious how to set this up made me finally do it, no more junk results (well less junk..). Thanks!
- dawnerd 5y agoFYI whoever made this, you can create clickable links to import filters. For example: https://subscribe.adblockplus.org/?location=https://raw.githubusercontent.com/dawnerd/ublock-filters/master/sites/google.txt&title=dawnerd%20-%20Google https://subscribe.adblockplus.org/?location=https://raw.gith... Quick edit: I know the domain is ABP but ublock origin picks it up.
- Quenhus 5y agoThat's actually really cool, thanks! However, the link don't work from HN for me. For the link to work, users need to click from a trusted domain listed here https://github.com/gorhill/uBlock/blob/bba4732c6b47134c3f54eec4e52e45b5b38f0429/platform/firefox/manifest.json#L57 https://github.com/gorhill/uBlock/blob/bba4732c6b47134c3f54e...
- gorhill 5y agoFrom 1.41.5b2 and above, it works on non-"trusted" sites with right-clicking the link, there will be an entry on the contextual menu to import the list. For older versions of uBO, you can already use the old way: abp:subscribe?location=[...]
- aspenmayer 5y agoHey, it’s the creator! Thanks for uBO.
- Quenhus 5y agoThanks for your incredibly fast response and help. (Do you have a notification when your username is invoked in HN?) And of course, thanks for uBO! In my case, the issue is that GitHub doesn't allow the apb|ubo protocol in links. However, no problem, I can use the method with "subscribe.adblockplus.org". Yet, subscribing from contextual menu is a great feature.
- rirze 5y agoNot unless you have both enabled... then ABP picks it up
- Melatonic 5y agoAnybody know of a way I could bulk import these into NextDNS?
- alin23 5y agoNextDNS is for blocking the site when you access it. It won’t hide it from the search engine results. You still need a browser extension adblocker (uBlock, AdGuard) that can modify the contents of any webpage for this to be effective.
- akselmo 5y agoOh sweet, thank you. Been looking for something like this.
- xahrepap 5y agoI know there’s some serious chicken/egg issue here, and solves a different problem than Google or DDG. but I think what I need is a way to “Like” or “Favorite” a page I’m on (maybe just a bookmark?) and that triggers my personal search indexer which will then index that page. Then maybe optionally “follow” other people’s favorites as part of your own results. Imagine someone crawling Twitter or SoF and you just “follow” their index and it gets merged with your results and disable/enable them for specific searches. I’ve been thinking about it a lot lately because I often am trying to remember something that helped me or I wanted to remember. But I can’t find it in my FF history. And Bookmarks feel too clunky.
- Gigachad 5y agoIn the end there is no perfect solution because the whole thing is gamed too much. Eventually spam sites will start paying popular users to like their spam page.
- Buttons840 5y agoThis attitude is too defeatist, and I'm not convinced. The OP's suggestion was that people would choose which other "favorite lists" or "indexes" they follow, this implies these are public. Who's going to follow someone who puts a StackOverflow clone filled with ads on their list? And if they do, and it shows in my results, I should be able to discover that someone I'm "following" has been compromised, and I'll unfollow them.
- Gigachad 5y agoThis would have been said a million times for social media and yet most large personalities are filled with paid sponsorships and no one really cares.
- RileyJames 5y agoAre large personalities not “unique”, and followed for the uniqueness of their character? Where as the “moderator of a filter” is followed on the basis of their output, the filter itself. I don’t disagree with your statement, but I don’t find it a compelling counterargument to the suggested solution.
- 0xTJ 5y agoI'm absolutely going to try this. The amount of time that I get blatant copies of Stack Overflow posts, only with the formatting stripped, is infuriating.
- kevin_thibedeau 5y agoI'm seeing a lot of Github clones on DDG lately. Some even stealing the GH favicon. I can't fathom what their goal is other than to inject exploits into devs who are high value targets. You'd think a responsible search engine would be culling these for the sake of general security.
- nerdbaggy 5y agoI’ve been seeing that a ton on Google as well. It seems to be the GitHub topics
- hda2 5y agoTwo important questions I don't see answered in the readme: 1. What is the criteria for a github copycat? 2. What is the process for having a website removed? I ask #1 because many businesses use their own git-hosting solutions to host their code but also use github as a mirror. It would be very easy for a competitor to get the website of a rival business listed if the criteria is not strict and specific enough. I recommend never blocking the entire domain unless the website is a repeat offender (to avoid mistakenly harming innocent businesses and the liability that it may cause). I ask #2 because many of these domains will likely be registered by honest people once the scammers are finished with them. There needs to be a way for the new owners to get their new domains delisted.
- aniforprez 5y agoFor #2 I assume a PR to remove the site could be sent and merged For #1 it seems to mostly be a list of SEO gaming sites that I've personally found to be supremely irritating and deserve to be in the list. They basically just mirror stack overflow and GitHub issues and provide obscured links back to the original source to make sure people stay on their site. You can peruse the list in the code yourself. It's just a text file
- hda2 5y agoI don't like low effort clones either. My only concern is the susceptibility of this project to being weaponized by dishonest actors. Some of us have been on the receiving end of false-flag spamming campaigns by competitors and have been penalized unfairly for it. If these filter lists take off, and I hope they do, they must have a well-documented process for accepting patches and another for handling malicious actors.
- metadat 5y agoBecause filters will fix the Broken Google problem?
- scim-knox-twox 5y ago???
- sam0x17 5y agoIt's crazy the community has had to come together to create rag-tag tools like this to patch up what is supposed to be a trillion dollar search engine that should be the pinnacle of human civilization. But then I remember it's the same sort of crazy whereby the richest country in the world has homeless people, child hunger, and decaying infrastructure :/
- tobr 5y agoCould you elaborate a little on the similarities you see between SEO spam and child hunger?
- atoav 5y agoIn capitalism rich/powerful entities don't necessarily have incentives to make the world (or even their own direct neigbourhood) a better place, and most big entities act only based on their incentives, not based on morals and ethics. They however have strong incentives to invest all available resources into their own survival, prolong their own life and/or earn more money. Depending on how the managment is in turn incentivised they will typically also prefer short term success over decisions that would make more sense long term — e.g. short term a search engine saves money by not spending a ton of money on quality when they are in a monopolist position, while in the long term it fould bite them.
- DaltonCoffee 5y agoBoth are unnecessary and confusing to see when set against the massive success of their creators.
- sam0x17 5y agoIn both scenarios you have a massive entity with near-infinite resources acting like it's so underfunded it can't take care of it's own shit.
- merpnderp 5y agoIt’s not a lack of resources it’s a lack of will. You can’t force parents to use the food programs. Hell around where I live you can’t stop the parents from stealing from their kids to buy drugs. Send a coat home with a kid on Friday, Monday they show up with no coat.
- a1371 5y agoHadn't thought about using uBlock Origin for this. I'll use the same technique to filter out Pinterest too then.
- theyeenzbeanz 5y agoOften enough I’ll see 5+ sites at the top of results for windows errors and they’re basically all trying to sell me golden hammer software to “fix it”.
- i13e 5y agoHow do I convert these filters into ABP-compatibile syntax? The adblocker in Brave's browser only reads that for some reason