12 ms·
Ask HN: Why do search engines not let you blacklist spam domains?
Whenever I'm searching for anything even mildly off the beaten path, it's not uncommon for the top results to be SEO stuffed spam websites, or maybe even real websites that I can't access (like paywalls or requiring adblocker exceptions to proceed). Usually pages from the same domains are top-ranked for other related searches too.
As a user I'd love to be able to tell my search engine to "Never show me results from this domains" (similar to blocking an account on Twitter) – but as far as I can tell there is no way to do this in either Google or DuckDuckGo search.
This seems like such low-hanging fruit to me that I'm wondering if other people have ever wanted this, and if there's actually a reason not to do it.
- fxtentacle 5y agoIn my opinion, most mainstream websites are spam by now. I understand why Google won't let me vote on the issue because they'll surely not like the results ;)
- elmerfud 5y agoIt's a simple answer because you don't pay them for that. They run a balancing act of providing you useful results while also surreptitiously shoving trash, that they actually make money off of, at you and hoping you won't notice or complain.
- PaulHoule 5y agoIf the search results were perfect you'd never look at an ad.
- lijogdfljk 5y agoA good argument for some distributed foss search then i guess. Wonder if it could ever be made to have search results return reasonably fast in a distributed large index?
- beamatronic 5y agoSometimes I am searching for information, and I want facts. Sometimes I am researching a purchase, and I want the sellers to compete for my purchase. Perhaps it makes sense to give search engines more context, such as, whether your intent is to spend money or not.
- dharmab 5y agoKagi will be paid when it exits beta and has this feature!
- jbverschoor 5y agoWhich means you can never use it in an incognito session without logging in
- dharmab 5y agoAn incognito session provides privacy against other people who have casual access to your computer. It provides no extra security or privacy in the context of search providers.
- CWuestefeld 5y agoDisagree. When the FBI comes to them asking for a list of everyone who's done a search for "pipe bomb", your anonymity would have been welcome.
- dharmab 5y agoKagi doesn't log much identifying information, so the government would have to compel them to change their system with a national security letter LavaBit style. https://kagi.com/privacy https://kagi.com/privacy
- freediver 5y agoWhile there is certainly possibility of FBI asking for this, I am afraid we couldn't respond to such request. Reason is simply we do not log or associate searches with an account. Our business model is selling subscriptions, not data. (Kagi Founder here)
- kelnos 5y agoI've been using Kagi for the past couple months (and love it so far!), but whenever I open a private browsing window, I revert back to DDG. Even though you say you don't associate search queries with accounts, it's hard for me to get over the "mental hump" to truly feel that in my bones and feel comfortable entering a "private" query while being signed in. I know it's not entirely rational. I mean, maybe it is a little bit rational: I only have your word (some random person that I don't know) that you don't -- and, critically, will never -- log queries associated with account. But regardless, it's hard to feel private when I'm entering something into a form on a website where I'm signed in as myself.
- rbinv 5y agoGoogle used to have that feature: https://www.ghacks.net/2011/03/10/google-adds-block-all-domain-results-to-search/ https://www.ghacks.net/2011/03/10/google-adds-block-all-doma... In my opinion, back then, they needed the data as a training set for spammy domain detection. Now that SERP spam is no longer a serious issue (in Google's eyes anyway), why bother. Google always knows what's best for you.
- Arnavion 5y ago>Now that SERP spam is no longer a serious issue (in Google's eyes anyway) Given the number of clones of StackOverflow and GitHub that show up on the front page, even at the cost of replacing the original SO and GH links that they copied from, I can only assume either Google's eyes are blind or the search engine devs are so good at their job that they never need to Google anything themselves.
- tetsusaiga 5y agoAnother fun one— next time a public figure dies, especially a slightly more obscure one, just google them/their death. Whole first page is just poorly written fiverr style articles, all by people for whom English is clearly a second language, at best. Eventually, legit stuff gets on the first page too, but theres a few instances where weeks later the first result is still one of these garbage seo-torture sites. Has really made me lose my faith in google lately.
- dreamcompiler 5y agoI'd be surprised if Googlers don't have an internal tool to filter out the crap that only they can use.
- sio8ohPi 5y agoAsked last week: https://news.ycombinator.com/item?id=30733227 https://news.ycombinator.com/item?id=30733227 TL;DR: No.
- ncann 5y ago
- sharps1 5y agoIf I had to guess, it's a lot of data they don't want to store. Current solution is either ublacklist, or add filters to uBlock Origin. Both are linked in the thread https://news.ycombinator.com/item?id=29546433 https://news.ycombinator.com/item?id=29546433 https://github.com/h-matsuo/uBlacklist-subscription-for-developer https://github.com/h-matsuo/uBlacklist-subscription-for-deve...
- ncann 5y agoData storage can't be the reason. No matter how you implement it, the storage required is peanuts compared to things like Youtube/Drive/Google Photos.
- DanAtC 5y agoKagi allows you to weigh domains.
- shafyy 5y agoI've also switched to Kagi recently and I think I find myself almost never doing !g (compared to DDG). And if I go to Google, it's mostly for hyperlocal results.
- Doctor_Fegg 5y agoSo far I've only found two areas where Google beats Kagi. One is, as you say, local/POI search. The other is deep diving into family history stuff - I've been tracing a 19th century composer and Kagi didn't surface most of the interesting documents. But for pretty much everything else it seems better than Google.
- freediver 5y agoThanks for the feedback! (Kagi founder here) Local results are on our roadmap, we should be launching it within few weeks. I am wondering if can you report problematic queries to https://kagifeedback.org https://kagifeedback.org so we can take a look?
- dharmab 5y agoIncluding fully blocking them if desired!
- qwnp 5y agoAlso you can completely block domains, and pin to the top of the results. I just started using Kagi, and they definitely seem to be doing a lot of stuff right.
- Gys 5y agoI switched to Kagi yesterday and I am so far very impressed. I did this search to answer another comment and later did the Google one: https://kagi.com/search?q=What+does+that+site+make+you+drag+the+text+when+you+try+to+highlight+it%3F https://kagi.com/search?q=What+does+that+site+make+you+drag+... https://google.com/search?q=What+does+that+site+make+you+drag+the+text+when+you+try+to+highlight+it%3F https://google.com/search?q=What+does+that+site+make+you+dra... Kagi shows the exact answer in the first result. Google gives no relevant answers on the first five pages (did not go further). Probably because Google focusses on 'highlight' too much. Kagi seems very aware of the context.
- drunner 5y agohttps://news.ycombinator.com/item?id=30374905 https://news.ycombinator.com/item?id=30374905 which in turn points to https://github.com/quenhus/uBlock-Origin-dev-filter https://github.com/quenhus/uBlock-Origin-dev-filter Doesn't answer you question. I wish google would use their enormous resources to fight this copy spam (and maybe its so hard that they are but you can't tell).
- MiddleEndian 5y agoGotta do it on your end. The uBlackList extension works for several search engines in Firefox and Chrome.
- krono 5y agouBlock Origin static filters to the rescue! Block results from specific domains on Google or DDG: google.*##.g:has(a[href*="thetopsites.com"]) duckduckgo.*##.results > div:has(a[href*="thetopsites.com"]) And it's even possible to target element content with regex with the `:has-text(/regex/)` selector. google.*##.g:has(*:has-text(/bye topic of noninterest/i)) duckduckgo.*##.results > div:has(*:has-text(/bye topic of noninterest/i)) Bonus content: Ever tried getting rid of Medium's obnoxious cookie notification? Just nuke it from orbit on all domains: *##body>div:has(div:has-text(/To make Medium work.*Privacy Policy.*Cookie Policy/i))
- Inviz 5y agowell that leaves blanks, right?
- krono 5y agoNo, these completely remove any matched results
- smusamashah 5y agoIf first page is all filtered links, we will get an empty first page.
- Kiro 5y agoYes, and creates blanks. Not visually but where it would be 10 results there are now 9.
- rg111 5y agoI received an ivite withing half an hour. So, I guess it varies.
- samcrawford 5y agoFiltering out the spam results is only half the problem. In my experience, a legitimate site's content is cloned by a spam site, and that one appears in a Google search and the legitimate one does not. The example that keeps hitting me is GitHub Issues. Filtering out the spam only removes the clones; it doesn't get the good results back in.
- lordgilman 5y agoI've been using these blacklists with uBlacklist to great effect: https://github.com/arosh/ublacklist-stackoverflow-translation https://github.com/arosh/ublacklist-stackoverflow-translatio... https://github.com/arosh/ublacklist-github-translation https://github.com/arosh/ublacklist-github-translation
- throw_m239339 5y agoBecause ultimately there is a conflict of interest between you as a search user, and google actual customers, advertisers who are often spammers themselves. Someone needs to pay for google search and since it isn't you... Google might have been better in the past, but since there is absolutely no serious competition whatsoever from a market perspective, Google technically doesn't really need to care about the quality of its search results anymore, only maximizing profits.
- deleted 5y ago[deleted]
- shrikant 5y agoFor the same reason that streaming services don't really give you the ability to filter by cast/crew or hide stuff you've already seen: to gently guide you into avenues that are more profitable for them, regardless of what you say you really want.
- btgeekboy 5y agoI just tried searching Netflix, HBO, and Prime for Bruce Willis; all showed me his movies. Is that not the filtering you’re expecting?
- lostcolony 5y agoThat's a fuzzy search. If I search on Netflix for "Bruce Willis", the first ten or so items are movies with him. Then there are miscellaneous thrillers and whatnot; unless I look at the details I don't know that they don't have Bruce Willis in them. A filter gives me a binary outcome; it either has Bruce Willis, or it doesn't. All of which is sort of the ops point I think; the search engine is fuzzy because it wants to show you many things for profit reasons. I don't think it's geared toward maximizing profitability at the expense of what you're looking for, exactly, so much as engagement as a proxy for profitability. The more you use the service, the stickier you likely are as a paid subscriber, so they'll happily shovel things that don't match the search (but are similar!...and that quickly jump the shark into being quite dissimilar, but hey, maybe!) to try and keep your eyeballs. A better demonstration of this handling is searching for movie titles they don't have. "The Princess Bride" on Netflix, for instance, doesn't even tell you "We don't have that", but instead "Explore titles related to: The Princess Bride". And while the first suggestion, The Neverending Story, is kinda a fair suggestion, some slightly later ones, like Top Gun and Zoolander, feel like a stretch.
- klibertp 5y agoI suspect the GP thought about something more complex, like "a movie directed by X or Y between year XXXX and YYYY starring Z but not V in the genre ... with tags ... but excluding ones tagged with ..." You can probably do this on IMDB or something similar, but then you can't watch it there. I'm largely guessing - I don't watch movies, but I read a lot of manga online, and the only site that allows similar queries and still has UX a bit better than late '90s (ie. mangaupdates.com) is a fan-made (ie. pirated) one with porn/hentai...
- beamatronic 5y agoIf search is free, then you are the product. Now if there was a paid search engine, and you were the customer, you would expect customer service, customization, no ads, privacy protection, etc.
- ncann 5y agoNo ads/no privacy/no customer service are fair game for a free service, but there's no reason why it should include no customization as well.
- g105b 5y agoI wish I could do this to tell DDG that I don't want to see any amazon.com products in my listing. I fell out with Amazon years ago and have been shopping independently since, but they have a stronghold over search engines with their out-of-stock listings.
- birthdaydog 5y agoI think people are right that the motivation is profit, but I'm not convinced it's to manipulate you in any way. I don't think driving you towards SEO'd blog spam is really all that profitable to google. My guess is that it's because it's an abusable feature, and that means hiring human moderators for it.
- lprubin 5y agoHow is it abusable if you're only blocking the domains for yourself?
- aaomidi 5y agoThe second this is added, the spam domains will just multiply. User interventions with spam doesn't work.
- Retric 5y agoDomains aren’t free the way sending email is. So, forcing more domains would be a significant expense that would seriously discourage these companies. Also, they would be easier to detect.
- charcircuit 5y agoIt practically is. You just add a new record to create a new domain.
- ColinHayhurst 5y agoBecause Google and Bing (therefore Duck) are increasingly answer engines, keeping you on their page, supporting the SEO ecosystem and most importantly their ad revenues and network customers. We and the other few independent search engines have not made enough dent on the market to suffer SEO spam. We'll have a way to deal with it (watch this space). Right now you'll certainly get results "off the beaten path" and with one click you can try out 8 other search options [0]. [0] https://blog.mojeek.com/2022/02/search-choices-enable-freedom-to-seek.html https://blog.mojeek.com/2022/02/search-choices-enable-freedo...
- mdasen 5y agoNot only are they becoming answer engines, in Google's case there's a decent chance that they're making money off the spammy pages. If the top results are "SEO stuffed spam websites", they're probably also loaded with ads. If they're loaded with ads, there's a good chance that they're using Google's ads. If the top results are "SEO stuffed spam websites", there's a good chance they're chock-full of affiliate links. If a search for "best baby formula" is going to end up costing Amazon an affiliate fee via an "SEO stuffed spam website", Amazon might as well just buy a Google ad for that keyword and cut out the "SEO stuffed spam website". If the results go to pages that are just going to cost a seller money anyway, it gives the seller more incentive to buy ads since they aren't getting free traffic from the search engine anyway. While being an answer engine keeps you on their page longer, feeding you SEO spam also keeps you coming back to their page; feeding you SEO spam signals to potential advertisers that they won't be getting free traffic from the search engine so they might as well pay for the traffic via ads. I'm not suggesting that it's a conspiracy to send you bad results, but it does seem likely that as long as they aren't losing traffic to competitors, it might not be something that becomes a priority.
- ColinHayhurst 5y ago> If the top results are "SEO stuffed spam websites", they're probably also loaded with ads. If they're loaded with ads, there's a good chance that they're using Google's ads. Bingo. If you can't get clicks on AdWords get them on AdSense.
- 5y ago
- layer8 5y agoGoogle in particular doesn’t care about what features the small minority of power users would want. Otherwise they wouldn’t have removed so many over the years.
- throwaway158497 5y agoBecause no product manager has been able to push it through various layers of beauracracy (yet). Plus more customizations means, their ML based personalized SERP page has failed in understanding your intent/needs.
- rc_mob 5y agokagi does. i love it. whenever they move from beta to paid I'll be a paying customer
- ajsnigrutin 5y ago...because pinterest would lose all their pageviews.
- guelo 5y agoI want a search engine that let's me exclude sites that have ads. Or even just exclude javascript.
- heavyset_go 5y agoThe more time you spend looking for the result you want, the more ads you will see. Some of those spam domains are filled with ads, as well.
- baskethead 5y agoWhat if we all started clicking on every ad on every spam page? Would they eventually get kicked off all the ad platforms for fraud? The only reason why Google and Facebook work is because their algorithms believe that we only click on the ads that we're interested in. If we click on every single ad, we will completely break their system within 3-6 months.
- hedora 5y agoNo; your clicks would be labeled as fraud, but the page likely wouldn't.
- jolmg 5y agoAdding -site:baddomain.com still seems to work in both Google and DuckDuckGo. You should be able to include that in the URL template used for search so it gets added to all queries. You can build your blacklist that way. E.g. https://duckduckgo.com/?t=hk&iar=images&q=-site:pinterest.com+-site:flickr.com+%s As an aside, I haven't used Google in a while, and I find it interesting how the first page shows only like 5 results and at the very end. The rest of the page is widgets like "top stories" and "people also ask".
- sytelus 5y agoImplicit signals are far more significant than explicit. Search engine already know which links are not getting clicks and which links are causing re-queries and which links causes pressing of back button. People do dumb things if you ask them not to show results from a domain or even “report spam”. For example, Fox News and CNN will all be marked as spam millions times a day even after people are finding what they are looking for.
- kelnos 5y agoI've long wondered the same! I recently started using Kagi, and it does have that feature. I've already blocked several domains from search results there, and I think the results pages are better (for me, anyway) as a result.
- asab 5y agoI see lots of “google is evil” narrative here… in reality they could add a feature to collect spam flags and still disregard user preferences. It’s just that the data will probably not help them, since adversaries are much more motivated to manipulate it for SEO profit than the average user who is unlikely to repeat that search
- aaron695 5y ago
- darinf 5y agoNeeva has this feature. Configure sites you want to see more from and sites you want to see less from. Easy.
- reaperducer 5y agoSimple answer: Google doesn't make money from giving you the right results. Google makes money by keeping you searching.
- Randolf_Scott 5y ago
- dalf 5y agoSearXNG allows to remove domains: https://github.com/searxng/searxng/blob/d1334beb4fb16beec298f894d75e94b241499e06/searx/settings.yml#L188 https://github.com/searxng/searxng/blob/d1334beb4fb16beec298... As far I understand, an improvement is to allows an URL like https://raw.githubusercontent.com/h-matsuo/uBlacklist-subscription-for-developer/master/list.txt https://raw.githubusercontent.com/h-matsuo/uBlacklist-subscr... in settings.yml, so there is no need to update settings.yml. Disclaimer: I am one of the maintainers of SearXNG.
- charlieyu1 5y agoAs a returning Diablo 2 player, it is sad that websites with useful information from 20 years ago are gradually replaced by SEO optimised crap gradually