6 ms·

I reckon these days search is pretty difficult and everyone knows how to game it. I recommend using a search engine that lets you effectively change which sites

I reckon these days search is pretty difficult and everyone knows how to game it. I recommend using a search engine that lets you effectively change which sites are shown. You can do this with Kagi, or with Google's Programmable Search Engines - I'm sure there are more too. In particular I block Youtube, not because they aren't sometimes correct, but because I don't want videos polluting the regular results - it just takes too long to get info from videos. An ability to upvote results for a given query seems tantalizing but I bet it would be gamed too. The DIY approach seems to be the only tractable one. In my case I only only results from domains I believe are correct. The whitelist approach does have downsides. Usually I'll vet new potential domains through social means like Reddit and this site, rather than identifying them through the search results. I believe there's an inherent tradeoff between discoverability and the gameability of the results. Though I do sympathize with folks who reminisce about 2008 Google Search results, there were probably orders of magnitude less content out there and a complete ignorance to how valuable your place is on your business and thus no SEO. I also personally disagree that yt-dlp is the "correct" result for the average user when they search Youtube Download. I highly doubt the average user would know or care to use the command line. A website front end would be more actionable for them.

kristopolous 3y ago
I'm a big fan of the non commercial site search engines because of the gaming aspect. If you're not generating revenue from the clicks the game mostly goes away. I'm not saying people aren't entitled to make some money, but it clearly incentivizes user hostile behavior. Maybe make it an option because legitimate sites like journalism also use this model.
- Renaud 3y ago
  Subscription model like Kagi seems to work pretty well against gaming the results. Their only remaining incentive is to be good enough that people keep paying for the service.
  - Nextgrid 3y ago
    It works not because they're somehow smarter or have more resources than Google at detecting spam/SEO, it's because unlike Google (and other ad-supported search engines), they make money from result quality and have an interest in blocking spam. Google on the other hand makes money off ads (whether on the search results page itself or on the spam sites), so spam sites are at best considered neutral and at worst considered beneficial (since they can embed Google ads/analytics, and make the ads on the search results page look relatively good compared to the spam). Black-hat SEO has been around since the early days of search engines and they managed to keep it at bay just fine. What changed isn't that there was some sudden breakthrough in malicious SEO, it's that it was more profitable to keep the spammers around than to fight them, and with the entire tech industry settling on advertising/"engagement" as its business model, the risk of competition was nil because competitors with the same business model would end up making the same decision. The same reason is behind the neutering of advanced search features. These have nothing to do with the supposed war on spam/SEO, so why were they removed? Oh yeah because you'd spend less time on the search results page and are less likely to click on an ad/sponsored result, so it's against Google's interests and was removed too.
    - ec109685 3y ago
      Kagi works because there is no incentive for SEO manipulators to target it since their market share is so small. Super tinfoil hat to believe Google wants to send users to blog spam websites (e.g. beneficial to Google). Anytime there is money to be made, there is an effectively infinite amount of people trying to game the system.
      - lanstin 3y ago
        Google is a complex system so “want” can just include we are making money from the blog spam and while we don’t like it other things take priority over fighting it as effectively as we could.
      - ZeePelli 3y ago
        It's never tinfoil-hat to assume that a corporation is, at very least, making sure not to fight too hard against any activity that brings it more revenue.
ysavir 3y ago
> In particular I block Youtube, not because they aren't sometimes correct, but because I don't want videos polluting the regular results - it just takes too long to get info from videos. Funnily enough, lately I've been prioritizing YT videos more when searching. So many sites now are just regurgitated SEO farms with minimal quality, and easy to see why: it's minimal effort to produce and cheap to host. But making a video takes time and effort, so has a much higher barrier to use as a click farm. More than once when traditional search failed me, I went to YT and found some video from 2009 clearly and eloquently explaining what I'm looking for in detail, and without any distractions because the person authoring the video clearly didn't specialize in the media format or show interest in experimenting. I've found it to also be a better source when looking a product to buy. Want to know which fan to get? Turns out there's a channel from a dedicated guy who keeps finding ways to test different fans and their utility and with multiple videos demonstrating his approach and findings. The mainstream channels aren't all that useful, but there's a ton of "old web" style videos (some even recent) passionately providing details for almost anything you'd think to search. And they're a gold mine.
- robrenaud 3y ago
  Would a browser feature that skipped to the relevant parts of the video based on closed captioning and understanding search intent be useful? It seems like this would be a good way for Google to fight to stay relevant in UX vs having the chat bots just quickly spitting out a readable answer. Hunting through ad laden webpages is annoying. Seeking to the relevant section of the video is a solvable problem, especially for videos above some viewership threshold.
  - dcow 3y ago
    Didn’t Google try this already? It seems useful to me, at least. IMO the next frontier of search is not better hypertext, it’s podcasts, audio, and video.
  - nulld3v 3y ago
    I've definitely seen Google do this already: https://searchengineland.com/google-tests-suggested-clip-search-results-video-jumps-directly-answer-273488 https://searchengineland.com/google-tests-suggested-clip-sea...
teeray 3y ago
> it just takes too long to get info from videos. I can’t wait until video transcripts get fed into LLMs just to eliminate the whole “This video is sponsored by something-completely-unrelated, more about them later. What’s up Youtube, remember to like, share, subscribe… 5 entire minutes pass on similar drivel… the actual thing you want, but stretched out to an agonizing length”
- execat 3y ago
  You need SponsorBlock. Usually people leave a "highlight" marker which tells you where you're supposed to jump to. Along with the regular "This video was brought to you by <insert>VPN".
lamontcg 3y ago
> Though I do sympathize with folks who reminisce about 2008 Google Search results, there were probably orders of magnitude less content out there and a complete ignorance to how valuable your place is on your business and thus no SEO. That was a decade after Google was created and people certainly understood SEO and Google was constantly updating its algorithm to punish people who were trying to game the algorithm. The wikipedia page on "link farming" for example references it happening as early as 1999 and targeting SEO on inktomi: https://en.wikipedia.org/wiki/Link_farm https://en.wikipedia.org/wiki/Link_farm I remember some internal presentations at Amazon around ~2004 about how boosting Google SEO on Amazon web pages increased traffic and revenue (and Amazon was honestly a bit behind-the-curve due to a kind of NIH syndrome).
- bee_rider 3y ago
  At the time it seemed like Google was winning, though. SEO seems to have gotten really good, or maybe Google just gave up.
stevage 3y ago
I have a hard time believing it's so difficult for a search engine to distinguish between a credible, respected website that has been around a while with some generated garbage that exists to be a search result. We humans can tell them apart, so in principle, computers can too.
- Nextgrid 3y ago
  Yes, this should be table stakes for a classifier - a company with the resource of Google can definitely solve that problem if they weren't themselves in the business of spam (advertising) and benefited from spam sites (as they often include Google ads/analytics).
  - navigate8310 3y ago
    Google is quite quick in plugging holes in AdSense but AdWords.
  - ametrau 3y ago
    > table stakes Always “table stakes”. Do you think in buzzwords also? I’ve always wondered this. Or do you think normal words and then translate it into this bandwagoning / membership proving garbage ?
    - dang 3y ago
      Hey, please don't cross into personal attack on HN. It's not what this site is for, and destroys what it is for. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful. Edit: unfortunately your account has been breaking the site guidelines in a lot of other places too—here are some recent examples: https://news.ycombinator.com/item?id=38825624 https://news.ycombinator.com/item?id=38825624 https://news.ycombinator.com/item?id=38825543 https://news.ycombinator.com/item?id=38825543 https://news.ycombinator.com/item?id=38783196 https://news.ycombinator.com/item?id=38783196 We eventually have to ban accounts that post like this, so if you'd please stop doing that, we'd appreciate it. On HN the idea is: if you have a substantive point, make it thoughtfully; if not, please don't comment until you do.
dandrew5 3y ago
Google's PSE is neat but there isn't a good way to manage switching between them. They could easily add a little dropdown to let you select which one to use as part of the public link UI they provide for each one individually. Giggle[1] gives me this ability and I run it locally (alongside Kagi) for more specific things to target domain lists I've been building over the years. 1. https://github.com/dan-lovelace/giggle https://github.com/dan-lovelace/giggle

kristopolous 3y ago

I'm a big fan of the non commercial site search engines because of the gaming aspect. If you're not generating revenue from the clicks the game mostly goes away. I'm not saying people aren't entitled to make some money, but it clearly incentivizes user hostile behavior. Maybe make it an option because legitimate sites like journalism also use this model.

Renaud 3y ago
Subscription model like Kagi seems to work pretty well against gaming the results. Their only remaining incentive is to be good enough that people keep paying for the service.
- Nextgrid 3y ago
  It works not because they're somehow smarter or have more resources than Google at detecting spam/SEO, it's because unlike Google (and other ad-supported search engines), they make money from result quality and have an interest in blocking spam. Google on the other hand makes money off ads (whether on the search results page itself or on the spam sites), so spam sites are at best considered neutral and at worst considered beneficial (since they can embed Google ads/analytics, and make the ads on the search results page look relatively good compared to the spam). Black-hat SEO has been around since the early days of search engines and they managed to keep it at bay just fine. What changed isn't that there was some sudden breakthrough in malicious SEO, it's that it was more profitable to keep the spammers around than to fight them, and with the entire tech industry settling on advertising/"engagement" as its business model, the risk of competition was nil because competitors with the same business model would end up making the same decision. The same reason is behind the neutering of advanced search features. These have nothing to do with the supposed war on spam/SEO, so why were they removed? Oh yeah because you'd spend less time on the search results page and are less likely to click on an ad/sponsored result, so it's against Google's interests and was removed too.
  - ec109685 3y ago
    Kagi works because there is no incentive for SEO manipulators to target it since their market share is so small. Super tinfoil hat to believe Google wants to send users to blog spam websites (e.g. beneficial to Google). Anytime there is money to be made, there is an effectively infinite amount of people trying to game the system.
    - lanstin 3y ago
      Google is a complex system so “want” can just include we are making money from the blog spam and while we don’t like it other things take priority over fighting it as effectively as we could.
    - ZeePelli 3y ago
      It's never tinfoil-hat to assume that a corporation is, at very least, making sure not to fight too hard against any activity that brings it more revenue.

ysavir 3y ago

> In particular I block Youtube, not because they aren't sometimes correct, but because I don't want videos polluting the regular results - it just takes too long to get info from videos. Funnily enough, lately I've been prioritizing YT videos more when searching. So many sites now are just regurgitated SEO farms with minimal quality, and easy to see why: it's minimal effort to produce and cheap to host. But making a video takes time and effort, so has a much higher barrier to use as a click farm. More than once when traditional search failed me, I went to YT and found some video from 2009 clearly and eloquently explaining what I'm looking for in detail, and without any distractions because the person authoring the video clearly didn't specialize in the media format or show interest in experimenting. I've found it to also be a better source when looking a product to buy. Want to know which fan to get? Turns out there's a channel from a dedicated guy who keeps finding ways to test different fans and their utility and with multiple videos demonstrating his approach and findings. The mainstream channels aren't all that useful, but there's a ton of "old web" style videos (some even recent) passionately providing details for almost anything you'd think to search. And they're a gold mine.

robrenaud 3y ago
Would a browser feature that skipped to the relevant parts of the video based on closed captioning and understanding search intent be useful? It seems like this would be a good way for Google to fight to stay relevant in UX vs having the chat bots just quickly spitting out a readable answer. Hunting through ad laden webpages is annoying. Seeking to the relevant section of the video is a solvable problem, especially for videos above some viewership threshold.
- dcow 3y ago
  Didn’t Google try this already? It seems useful to me, at least. IMO the next frontier of search is not better hypertext, it’s podcasts, audio, and video.
- nulld3v 3y ago
  I've definitely seen Google do this already: https://searchengineland.com/google-tests-suggested-clip-search-results-video-jumps-directly-answer-273488 https://searchengineland.com/google-tests-suggested-clip-sea...

teeray 3y ago

> it just takes too long to get info from videos. I can’t wait until video transcripts get fed into LLMs just to eliminate the whole “This video is sponsored by something-completely-unrelated, more about them later. What’s up Youtube, remember to like, share, subscribe… 5 entire minutes pass on similar drivel… the actual thing you want, but stretched out to an agonizing length”

execat 3y ago
You need SponsorBlock. Usually people leave a "highlight" marker which tells you where you're supposed to jump to. Along with the regular "This video was brought to you by <insert>VPN".

lamontcg 3y ago

> Though I do sympathize with folks who reminisce about 2008 Google Search results, there were probably orders of magnitude less content out there and a complete ignorance to how valuable your place is on your business and thus no SEO. That was a decade after Google was created and people certainly understood SEO and Google was constantly updating its algorithm to punish people who were trying to game the algorithm. The wikipedia page on "link farming" for example references it happening as early as 1999 and targeting SEO on inktomi: https://en.wikipedia.org/wiki/Link_farm https://en.wikipedia.org/wiki/Link_farm I remember some internal presentations at Amazon around ~2004 about how boosting Google SEO on Amazon web pages increased traffic and revenue (and Amazon was honestly a bit behind-the-curve due to a kind of NIH syndrome).

bee_rider 3y ago
At the time it seemed like Google was winning, though. SEO seems to have gotten really good, or maybe Google just gave up.

stevage 3y ago

I have a hard time believing it's so difficult for a search engine to distinguish between a credible, respected website that has been around a while with some generated garbage that exists to be a search result. We humans can tell them apart, so in principle, computers can too.

Nextgrid 3y ago
Yes, this should be table stakes for a classifier - a company with the resource of Google can definitely solve that problem if they weren't themselves in the business of spam (advertising) and benefited from spam sites (as they often include Google ads/analytics).
- navigate8310 3y ago
  Google is quite quick in plugging holes in AdSense but AdWords.
- ametrau 3y ago
  > table stakes Always “table stakes”. Do you think in buzzwords also? I’ve always wondered this. Or do you think normal words and then translate it into this bandwagoning / membership proving garbage ?
  - dang 3y ago
    Hey, please don't cross into personal attack on HN. It's not what this site is for, and destroys what it is for. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful. Edit: unfortunately your account has been breaking the site guidelines in a lot of other places too—here are some recent examples: https://news.ycombinator.com/item?id=38825624 https://news.ycombinator.com/item?id=38825624 https://news.ycombinator.com/item?id=38825543 https://news.ycombinator.com/item?id=38825543 https://news.ycombinator.com/item?id=38783196 https://news.ycombinator.com/item?id=38783196 We eventually have to ban accounts that post like this, so if you'd please stop doing that, we'd appreciate it. On HN the idea is: if you have a substantive point, make it thoughtfully; if not, please don't comment until you do.

dandrew5 3y ago

Google's PSE is neat but there isn't a good way to manage switching between them. They could easily add a little dropdown to let you select which one to use as part of the public link UI they provide for each one individually. Giggle[1] gives me this ability and I run it locally (alongside Kagi) for more specific things to target domain lists I've been building over the years. 1. https://github.com/dan-lovelace/giggle https://github.com/dan-lovelace/giggle