8 ms·
Search results are so heavily weighted toward commerce, products and services because those are the people that can spend money on SEO. It's made the internet,
by ehnto 7y ago
Search results are so heavily weighted toward commerce, products and services because those are the people that can spend money on SEO. It's made the internet, seen through the eyes of search results, seem like an aggressive, shady market bazaar.
I try searching for anything remotely bike related, bike community, mechanical information, or just general cool bike stuff and I can't find the human community underneath all the fluff articles trying to sell me shit. The internet, as free as it is, has been overflowed with commercial activity. Which pretty closely reflects the real world, but damn that's a shame.
- jacquesm 7y agoOne nice solution would be a search engine that throws out anything related to commerce or linking to commerce.
- orkon 7y agoThen the search engine itself would be probably thrown away :-D
- SomeOldThrow 7y agoMy favorite parts of the internet have no relation to ads or commerce at all.... including wikipedia. When you’re looking for information commerce is a massive distraction.
- snlnspc 7y agosadly true, but an open source engine like searx could be made to do such a thing
- tomcam 7y agoHow could it be sustained? Search engines are expensive to run.
- pmoriarty 7y agoIt could be a subscription service.
- Chico75 7y agoThe real cost is in the infrastructure needed to run it unless you go full P2P
- edoceo 7y agoP2P it is then!
- CodeWriter23 7y agoYes, Search could use some Bittorrent-like disruption.
- MR4D 7y agoI wonder though - is too much of the P2P universe now mobile devices instead of regular computers that it wouldn’t be feasible?
- WrtCdEvrydy 7y agoP2P can be done on mobile but you're fighting against limited resources (storage, bandwidth, CPU). The reason P2P got good on PC is because storage got cheaper, bandwidth was increasing and CPUs kept getting faster.
- MR4D 7y agoRight. So texting on P2P is easy and appropriate, but I can’t see how building a search engine on P2P would really work well given those limited resources.
- mythrwy 7y agoIt's important enough, and currently wastes enough of my time (read: SEO crap, Medium and Reddit are top results for... everything) that I'd happily pay for a good search engine!
- ofrzeta 7y agoThere's YaCy already: https://yacy.net https://yacy.net
- alltakendamned 7y agoNot entirely throwing out commerce, but I sometimes use https://millionshort.com/ https://millionshort.com/ where you can remove up to the top 1 million websites from the results.
- rdiddly 7y agoJeez, that's surprisingly (and kind of tragically) effective!
- dorgo 7y agolol, if this becomes popular then everybody needs to reverse their seo efforts and optimize for position 1 million instead.
- jacquesm 7y agoThank you, I already knew about it but had somehow forgotten.
- jen729w 7y agoAmazing! This does something that I wish they would all do — allows me to block a site. I never, ever, for any reason, want to see results from the Daily Mail. I don’t even care if it’s the most relevant content: I just never want to see that site in my results. I know I can block it in hosts or whatever, but that’s not what I want. I don’t even want to see the link. I wish DDG would give me this option.
- rchaud 7y agoI feel the same about sites like inc.com, fastcompany.com, entrepreneur.com, forbes.com. They are the absolute worst, shallowest content for business journalism, but they're right at the top of search results, and their blogspam fills up my Pocket "Discover" recommendations.
- Anthony-G 7y agoHey, thanks. That's a great site! You get some very interesting search results and – even better – there's no third-party JavaScript or other trackers.
- swayvil 7y agoAgreed. Much like a spam-filter. Seems straightforward enough. What am I missing?
- Gene_Parmesan 7y agoWhat you're missing is the fact that larch search engine companies have no profit motive to do this. They get the vast amount of their money from commercial ad sales -- ads for commercial results. Why would they purposefully castrate their highest paying customers? And if you think the answer is to create and host a new, independent engine, you will be hard pressed to develop a good one, and to find the money to keep it running.
- swayvil 7y agoHow about a layer over Google? Take Google's search results and filter them. Any obvious barriers to that?
- tyingq 7y agoThey are pretty aggressive about detecting that and throwing up captchas.
- swayvil 7y agosee user smt88's idea above.
- jasode 7y ago>Any obvious barriers to that? Rate limits.[0] One can't really build a robust and comprehensive (server-side) search engine on top of Google's search API. In other words, Google Inc isn't going to provide an API so powerful that one can build an "alternative to Google" with it. On the other hand, if you're talking about a "client-side" filter instead of a new search engine, (e.g. a client-side webscraper that flips through the Google results pages), you'd run into multiple problems. E.g. random CAPTCHAS screwing up the automation script, and the difficulty of creating a "rules engine" (e.g. machine learning) on your local computer that recognizes blog articles with a commercial slant. [0] https://www.google.com/search?q=google+search+api+rate+limit https://www.google.com/search?q=google+search+api+rate+limit
- SomeOldThrow 7y agoThis is what happens when you don’t give users control over how search works. The weighting is blatantly user hostile by default because it’s so directed towards people who pay for your eyes. I would pay a massive amount of money for a search engine with a “no commercial results” switch. Of course paid search introduces class issues. I think we should view search like a public utility because it’s absolutely necessary for surviving the world.
- nitrogen 7y agoThe original designers of the web envisioned a world where every user could have their own chosen "agent" to navigate the web for them. This is where the User-Agent HTTP header comes from. There would be search agents, data mining agents, and interactive browsing agents. We missed out on like 90% of what the information revolution was supposed to bring to users, and instead the data giants have captured all the value.
- mostlyjason 7y agoWe expect everything on the web, including our user agent, to be free as in no cost. The only way to pay for the development cost is to sell to third parties aka advertisers. If people actually paid money to develop more advanced user agents there would be a market for them. Why do people prefer free over paid? I think people underestimate the influence of advertisers and underestimate the value of their own time spent researching and dealing with crappy products. It’s almost like a psychological bias. I like the idea of a better user agent but would I pay for one? Judging by my refusal to pay for quality journalism I’m guessing it’d be a hard sell. That’s another thing I should be willing to pay for but don’t for some weird reason.
- 1123581321 7y agoIf you use a Mac, DevonThink and its associated products are great examples of paid user agents that seek out and organize information for you. Their free trials are fairly generous.
- swayvil 7y agoOur culture is a frog nailed to the floor. The nail is commerce. The frog can still sorta function. Wiggle in little circles. It ain't pretty or healthy or free. No hopping. I wonder what it would be like to not have that nail through us.
- RestlessMind 7y agoCommerce -> revenues -> viable businesses -> sustainable jobs -> peaceful livelihood prospects. No commerce -> no jobs -> unrest -> violence and wars (citation: centuries of human history). Maybe, commerce is the nail which constrains humanity both in a good and a bad sense - by providing a peaceful alternative to violent lifestyles and by creating a consumerist culture?
- addicted 7y agoIt’s almost like there could be an inbetween because full throated commerce in every aspect of life and community and no commerce at all.
- itronitron 7y agoNo, commerce doesn't have to be a nail. It's just restricting the utility of the internet at this point, which is limiting commerce. The internet used to be an antidote to the shopping mall, but now it has become one.
- 50656E6973 7y agoFalse dichotomy. There is such a thing as too much commerce, as is evidenced by our current state of ecological self destruction
- hansjorg 7y agoIt might be possible to have both commerce and some form of democracy though.
- bradleyjg 7y agoWhen there's a trade that makes both sides better off that increases the happiness in the world. It's a Pareto improvement. When marketing convinces someone to buy something that ends up being a bad trade (i.e. the person that buys it is not made better off) then there is no such improvement and the world is not made better off.
- baxtr 7y agoI have made the same experience. Funnily enough, I started using the Russian search engine Yandex a while for these use cases. Results are not bad...
- userbinator 7y agoDon't forget the "showing results for Y, click here to search for X" when X is what you actually wanted to search for. It's ironic that, in trying to make search "human friendly", Google has also succeeded in giving it all the negative traits of a human --- the "human" that is more like a commissioned salesperson. I know the argument is that humans frequently make mistakes so "we should just show them what we think they wanted", but that's just opening the door for manipulation. Search was much better when it was closer to a "grep the Internet". IMHO machines should remain "dumb" (for lack of a better adjective) and leave the important decisions to the users, keeping the latter in control.
- phkahler 7y agoGrep the internet and sort by pagerank was great. Then everyone removed links that went offsite. Then link farms. The fake blogs and influencers are the evolution of link farms. Unfortunately sellers will game the system. How do we create a way for buyers to game it?
- tarsinge 7y agoMaybe it’s time to bring back curated web directories like the original Yahoo?
- marvin-83 7y agoYeah, I think so. Curation-as-a-service could even become a viable market. It's arguably already happening to a certain extent through various platforms, creators, and aggregators; but it's not really split off as its own service yet.
- inimino 7y agoPagerank was great when it was new because it had never been used before. Once Google got popular enough to be worth gaming, it became an example of Goodhart's Law. https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- 7y ago
- ackbar03 7y ago>"Which pretty closely reflects the real world, but damn that's a shame." This sums it up nicely haha
- robertAngst 7y agoI find some of my top 'competitor' blogs are worse quality, but somehow perform better due to having tons of links/content. Something like 'Save Money on Food', I am positive I am best-in-class, but I am only front page google, on occasion. Without a doubt, I should be the number 1 result, and instead, well SEO'd mom-science wins. Btw, does this mean google should be looking for crappy code/design=genuine website?
- Gene_Parmesan 7y agoThe catch of course is that anything search engines start looking for, SEO will start optimizing for. If crappy code == genuine website, then get ready for a shitstorm of shitty looking SEO-enhanced pages. Come to think of it, that might be kind of funny for a day or two.
- Waterluvian 7y agoHuh. Maybe we need two distinct search modes: library vs. bookstore. And not just the "shopping" tab but a fundamentally different set of algorithms for page ranking.
- ajmurmann 7y agoFor many topics I now go straight to search Reddit. It avoids all those articles that SEO the shit out of a topic where I was just looking for a two sentence answer. Edit: I'm often surprised how search engines fail to exclude obvious SEO farms. Especially looking at articles about stock, you frequently just find a ocean of what send to be clearly autogenerated articles.
- baddox 7y agoAny time I want to search reddit, I use Google and put “reddit” at the end of the query (I know about the site: query, but there’s really no point to type the 5 extra characters). Same tactic to find genuine content, but I find Google’s results to be better than Reddit’s own search.
- superasn 7y agoThough I'm no expert at this but these days negative SEO has also become a part of the search engine optimization. What this means is that some competitor can create thousands of bad links to your site causing your site to incur a penalty in search ranking. Google has created a disavow tool for this but for someone who runs a blog and not super tech savvy may not know about it and regardless it is still a burden for anyone to keep up with all this. Again, I'm not an expert at this and I'm doubtful that a billion dollar search engine can be tricked by such shenanigans but it may be something worth looking into. if your site traffic drops
- philippoi 7y agoI've been lamenting this demise of Google over the last year or two, but it's been especially foul the last few months. Similar to your biking anecdote, I was trying to find any simple trouble shooting help for a home coffee brewer that suddenly stopped working. I couldn't find any results in the first 4-5 pages that weren't trying to sell me a new machine. The manufacturer is also partly to blame, because I couldn't find anything remotely relevant either on the product page or in the PDF owner's manual. It's a shame what these tools have become when they could be so much more.
- FutureSpec 7y agoI find that hard to believe. I used the phrase "home coffee brewer stopped working" just now in Google and see nothing but troubleshooting articles and forums from the very first result onward.
- userbinator 7y agoIf you put in an exact model then I suspect the results will be very different. The other infuriating thing that's related is if you're trying to find a schematic or service manual, Google thinks you're just looking for the (often useless) user manual. Searching for the user manual does not make it think you're looking for the service manual... it's absolutely idiotic, because who would search for "service manual" but actually want a user manual? Beyond some weird conspiracy theory involving anti-right-to-repair, I can't explain why.
- hombre_fatal 7y agoI don't even understand the thread-OP's complaint. Googling "bike forums" gives me pages of what they say they can't find.
- jen729w 7y agoI’ve tried switching over to DuckDuckGo.com a bunch of times over the years, but this time the move has stuck. I think one of the reasons is that I do not get a bunch of the ‘smart’ stuff that Google tries to do for me. In a way it’s like using a search engine from 10 years ago ... and it’s better. There are things that I do miss — quick cards for things like currency conversion, flight details, that sort of thing - but the mighty Duck is getting better at those, and if I need to reach in to Google I can always just !g it.
- Lucadg 7y agoThere was a search feature in Google called "discussions" I think. You could find forums. IIRC then same was possible for blogs. Both were very precious and both are gone. There was no money in them I guess. This needs to be fixed.
- jddj 7y agoIn my recent experience, the only thing which causes me to tag a !g onto my duckduckgo searches is a need for recent results (as in, the last couple of hours -- for the results of a sports match for example). Anything else seems to be served by the other providers just as well/poorly, and occasionally better.
- ChuckMcM 7y agoPretty much captures it. That was the founding principle of Blekko which was that humans could curate a core set of 'good' websites for a topic and all the crap would not have a foot hold to show up on the page. So to share some of the challenges with that (if anyone out there wants to try again) they are as follows: * 'Search' means different things to different people, and we've been trained that a 'search engine' finds anything on the web (for the most part). The product Blekko built could more accurately be called a 'reference' engine which was used successfully by people trying to find facts or data and were not generally trying to find things to buy. * Have your own advertising system, all in house, where you don't have to "revenue share" with anyone if you don't want to. At its peak Blekko was serving over 10M queries per day which, if we had owned all the advertising revenue on those searches would have kept us going and growing. That said, building an advertising system is both difficult and fraught with patent risk / bad-actor risk. * Don't let anonymous users use the service. This is perhaps the hardest thing, most people won't give up an email address for even the most useful of services, however since you're spending money serving up search queries you don't want to waste that money serving up queries to bots and other bad actors. At any given time when I went through the logs there were between 5% up to nearly 18% of the queries were 'suspicious' or likely bots. That is 18% of your capacity you can't give to "real" humans if you can't control that traffic. * Build a relevancy ranking system rather than a popularity ranking system. Search results have two metrics of interest, precision and recall. For a reference engine you want to focus on precision over recall. And while existing search engines use the "virtuous cycle" of search & click to track popularity (which can be an indicator of precision but is better at indicating click-baityness) build your ranking engine using NLP based evaluations. * Your document index size should target 5 billion documents with a goal of 10 billion documents. Scale your cluster and algorithms to process a query to that index in 100mS or less. Do that and win :-) Or find the next barrier to creating a useful way to search the web for information.
- smsm42 7y agoI wonder if there's a ripe market for a search engine that specializes on "clean" search results. Google grew into its current titan status by, in part, providing the results with higher relevancy and accuracy than the competition, and avoiding the primitive SEO cheats of those days (like spamming the page with irrelevant keywords to capture traffic). Maybe that's the time for an engine that could avoid modern, more sophisticated SEO cheats.
- ben_jones 7y agoAnd it has resulted in communities like the ones you mention moving to aggregators like Reddit, Facebook, or even HN. And I’m ok with that - it’s like how communities IRL gather in churches, bars, or other areas. Why shouldn’t the internet imitate actual life?
- wvh 7y agoI've noticed the same with song lyrics when I try to find a particular song I've heard somewhere. Hundreds of results of the same handful of "top" pop music industry artists but anything alternative is hard to find, even if the search query very clearly is not a good match for those popular results. I know the stuff I'm look for is out there, but it just won't show up.