9 ms·

Hi Google, please stop pooping the bed: a desperate plea from the indie web

256 points by bwb 2y ago

jasonvorhe 2y ago
If you're still barking in the general direction of google and similarly sized search engines, good luck. I'd rather look towards kagi and brave.
- viraptor 2y ago
  I feel too that we're already past the point where they could fix things. They must be aware of issues and do the exact opposite of fixing them. It's a question how fast we can convince general public to move away, rather than when Google will improve.
  - jasonvorhe 2y ago
    I wouldn't give much regarding the general public. If you have a small blog they're usually not your target audience. Accept that the days of "making a few bucks off of advertising" on hobby publishing are over, focus on writing about things you care about without it reeking of being written by ChatGPT and chances are people will come at some point. If small publishers don't change their behavior in radical ways, nothing will ever change.
    - viraptor 2y ago
      > nothing will ever change. That's probably what MySpace people thought. And Friendster. And Altavista. And ...
      - jasonvorhe 2y ago
        I don't get what you're trying to say. I'm not arguing in favor of doing nothing in an ever-changing market, which is what Myspace and Friendster did most of the time. I'm saying the old world of "have good content/SEO and you'll see some traction thanks to Google" is dying and that you can either adapt or drown in a flood of LLM-created SEO blackhat nonsense. Google does not care about you and they've been in the "too big to fail" corner for too long. If you want content made by humans to continue to exist and want to see some of them be able to make a buck off of it, support smaller search engines and creative people and their publications.
        viraptor 2y ago
        I mean that services will come and go. Some will stay for longer than others, but even without a coordinated action, Google may go away. It's also losing to Perplexity and ChatGPT with many people. And many geeks are very vocal about their love for Kagi. (Including me) Things may change over time and may not be too big to fail.
- wbl 2y ago
  And how is the experience? Even with Bing I find I have to switch to Google to get useful results for my queries.
  - senko 2y ago
    Kagi user here, I almost never need to !g. Used DDG a few years back and stopped after I had to fallback to Google for most qs.
  - girvo 2y ago
    Google is so odd these days, its like there are two "streams" of results. One is the kind of useless slop that the article in question is rightfully complaining about, but other results are still useful and high quality. The second stream is what I seem to get when I'm forced to "!g" when I'm using DuckDuckGo, and I can't work out why theres this stratification, this separation. Years ago, it was very rare to get crap results from Google. Now its far more common.
  - alpaca128 2y ago
    It took me about two weeks until I got consistently good results with DDG, it often just takes about one more search term because it doesn't try to (often incorrectly) guess the context based on search history like Google does. That was years ago, since then DDG has gotten better and Google a lot worse. I could understand your complaint if it was about image search, but I don't see how anyone can look at today's Google results and think they're still superior.
  - viraptor 2y ago
    Kagi is simply better in every way I can think of and I've been a paying user for over a year now.
  - justinclift 2y ago
    Haven't needed to use Google at all since using Kagi several months ago, whereas when using Duck Duck Go for a few years (prior to Kagi) I'd (very occasionally) need to.
- lolinder 2y ago
  Kagi is a viable choice for individual people, but it's not an answer for the concern of how to get traffic to visit your site and click on your affiliate links (which is what TFA is about). Kagi works well for users precisely because it's small enough that no one benefits from gaming its algorithms.
  - rockskon 2y ago
    How is Kagi faring after Google's exclusivity deal with Reddit?
    - lolinder 2y ago
      Um, I don't know when that would have kicked in because I haven't noticed a difference. Reddit still shows up whenever it's relevant, and the Forums lens turns up even more Reddit if you're worried you're missing any on the initial search.
    - klooney 2y ago
      I think Kagi uses Google's APIs as a part of their mix of search results.
  - Ferret7446 2y ago
    Why is it Google's job, who TFA isn't paying AFAIK, to get traffic to visit their site?
    - lolinder 2y ago
      I don't think it is, but recommending Kagi as a solution for their complaint misses their point.
jqpabc123 2y ago
Simple answer --- stop using Google. Get your friends and family and other internet users to do the same. The only thing likely to get their attention is if enough people follow suit.
- 23B1 2y ago
  "Simple answer, just get everyone to stay home for three weeks and the virus will die out" "Simple answer, just stop electing corrupt politicans" "Simple answer, just stop war" k
  - deleted 2y ago
    [deleted]
  - senko 2y ago
    Subscribing to Kagi is definitely easier (and less costly!) than handling the pandemic, ensuring world peace and utopian government. Edit to add: and since we're talking about indie web: https://search.marginalia.nu/ https://search.marginalia.nu/
    - nemomarx 2y ago
      Me subscribing to it is. Getting everyone I know to subscribe to it is at least a step more complex than getting people to move to Firefox, and I haven't quite worked out a strategy for that yet.
      - justinclift 2y ago
        Search results being absolutely shit is probably more of a problem for people than their choice of browser. When someone complains to you directly about Google search results being shit, maybe point them to Kagi and mention it has a free trial account (100 searches still?) that's not time limited. That way they'll have a solution they can turn to the next time Google pisses them off significantly. :)
        2y ago
fny 2y ago
I find it ironic that there’s a desperate plea for the indie web from a Substack blog.
- croes 2y ago
  To get more visibility?
  - CSSer 2y ago
    I think the irony works either way.
  - CharlieDigital 2y ago
    If OP was going for visibility by posting it HN, Reddit, etc. then does it matter if it was Substack or his own blog or even the Shepherd site?
    - johnnyanmac 2y ago
      Technically, yes. If the thesis is "Google search is inaccurate and overrun with SEO optimized slop and ads", them using a blog utilizing that same SEO optimization feels a bit ironic. Feeding the beast you want to change.
      - bwb 2y ago
        I don't quite understand, I still want Google traffic, they have a monopoly. Nothing has changed there.
  - bwb 2y ago
    That is a private blog, so I don't care if anyone sees it. I just wanted something I updated every 2 to 3 weeks with my frustrations, wins, and so on as I build Shepherd.com. I was on Twitter's free email thing but they shut down and imported to Substack. I just didn't want to pay for anything as the only person who reads it is me and my mom. It has 100 readers on it, as its just for me really.
- 2y ago
deleted 2y ago
[deleted]
grugagag 2y ago
Google didn’t shit the bed, they just did whatever was profitable to them and they’ll continue doing so until they either fail or there’s no more profit to be squeezed. They dropped the “don’t be evil” motto a long tome ago.
skybrian 2y ago
It looks like shepherd.com is a book review site that doesn't have any reviews, just ratings. It links to Amazon. Is this really the sort of content Google should be returning? Edit: it seems I missed the link to the actual book reviews because the link text is uninformative: "Chosen by 1 person - see why." (Sometimes it goes to reviews, sometimes not.) And the word "review" never appears on the pages that have reviews. Seems like bad SEO? If you're looking for book reviews, here's a website with some pretty great reviews: https://www.thepsmiths.com/ https://www.thepsmiths.com/ (Content warning: the authors are conservative.)
- passwordoops 2y ago
  Just went over there and yes they have reviews from readers https://shepherd.com/book/exit-strategy https://shepherd.com/book/exit-strategy
  - skybrian 2y ago
    Thanks, I missed that.
- bwb 2y ago
  Hi Brian, we don't have reviews at this stage; we have purely positive recommendations. And we push readers toward a very specific format for why the book resonated with them (Gestalt psychology so you see it through their eyes). Given the nightmare that is Goodread's unmoderated review sludge, I wanted to focus on the positivity and why someone loved a book, so we are uniquely focused on highlighting only books that readers love. In 2025 I will be adding "reviews" of any books using our Book DNA review format (which tries to narrow in on why someone liked/disliked a book so we can match you with people who share your Book DNA). A big part of my mission is to focus only on books that people love and positivity.
- willywanker 2y ago
  Was that content warning really necessary, or do we also tend to use them for liberal websites (for whatever definition of liberal) ?
23B1 2y ago
There are people inside Google who can put a stop to this, if they merely stand up and say "this is wrong" including their CEO. It simply requires bravery and faith in Google's institutional ability to continue to profit doing different things. Big companies have pivoted before on the heels of brave executives. This applies as much to search as it does to privacy, AI, their transparency, their support of open source, and their weaponization of their browser. It's a crying shame to see how enshittified a company that could be changing the world has become.
- immibis 2y ago
  And a willingness to be fired the next day. Even the CEO can be fired.
  - mcmcmc 2y ago
    So it’s the shareholders and the board who are at fault then. I’d posit that anyone holding Google stock is part of the problem and responsible for their continued pattern of anti-user and anti-consumer behavior.
    - DrillShopper 2y ago
      It's actually late stage capitalism that is at fault. Line must always be going up, and if it doesn't go up as fast as someone thought it would then we make it go down. If line goes sideways we sell and make it go down. It must go up. Always up. Or it will go down.
      - mcmcmc 2y ago
        You can’t blame the system and pretend none of that lands at the feet of people willingly engaging with it for their own benefit.
        DrillShopper 2y ago
        Some of it absolutely lands at the feet of the people willingly engaging with it, but there are far, far more people who are caught up in and and only go along with it to have a roof over their head. I don't think the burger flipper at McDonalds, the stocker at Walmart, the barista at Starbucks, the bartender at the bar down the street, the person working the register at the gas station, or the picker at Amazon (just to name a few) share in this blame. They're all trying to eke out an existence for themselves and their families.
immibis 2y ago
Hi (company that benefits from hurting me), please stop hurting me. Company: No.
niobe 2y ago
8-10 years google search was amazing. A well crafted query would hit informational gold most of the time. I've been noticing and commenting on the decline privately for most of the period, but it's only in the oast year this seems to have come to broader awareness. There's an argument that in an information economy, searching for information should be treated like a public utility necessary for the functioning of society. I'm not making that argument but when you experience the long slippery slope of degradation of a service that was near ideal for the technology of the time, it does xome to mind. That's the thing. It DID work. Really well for a while. But it was always atomic and context-less. We now have the opportunity to make it even better by refining results through dialogue. I hope someone does.. soon.
- zx8080 2y ago
  > We now have the opportunity to make it even better by refining results through dialogue. I hope someone does.. soon. TBH, I don't want dialog (too long and slow, leave dialogs for human to human comms), I just need my query words to exist in the result pages. Nothing more. Google messed this up long time ago.
  - throwaway48476 2y ago
    I wouldn't mind a dialog that helps me build an advanced query.
    - nradov 2y ago
      Have you tried advanced search? https://www.google.com/advanced_search https://www.google.com/advanced_search
      - throwaway48476 2y ago
        I'm aware of advanced search. I'd like it to be more user friendly and widely used. I think a chat bot would help.
  - munificent 2y ago
danpalmer 2y ago
I have no specific information here, never seen/used Shepherd until I saw a list on there a few days ago. Disclaimer, I work for Google, but not on anything related to this space, and this is based on my previous job where we did some SEO for an ecommerce site. The example list given just looks a lot like spam when you squint. It's a list of affiliate links to buy products, and there are many HN threads talking about the abundance of affiliate link aggregators being a blight on the web. The commentary does look useful, but distinguishing between good commentary and bad commentary is hard, whereas distinguishing between a site designed to extract affiliate commission vs one more about the content is easier. The comparison given to the other results here is frustrating, I know, but probably not a valid experiment. All the major search engines change results based on the user using them, or the IP address, or the region, or whatever, so it's impossible to know what others see. The developer of a book-focused shopping site is likely to get very skewed results for a book related query. My results were noticeably better. The author says that a Bookshop.org list they created that links back to Shepherd is ranking #2, and this kinda makes sense to me. Bookshop.org sells the books, it makes sense that would rank above a site that only links to (and makes money from) sites that sell books. SEO, and people getting annoyed at not ranking, has been a thing for 25+ years, I don't think this instance is any different.
- smallerize 2y ago
  (That's not what dearth means.)
  - Jerrrrrrry 2y ago
    portmanteau of "dithering" and "blight" and "earth"
  - danpalmer 2y ago
    Whoops, thanks.
- strunz 2y ago
  Do you use Google anymore? It has gotten so bad, I regularly can't find things on it anymore, no matter how I search for things it will keep showing the same results. You seem to have blinders on. How do you explain the site getting less and less traffic over time?
jeffbee 2y ago
I've had my lifetime quota of hearing the rants of guys who think their search quality metric is objectively superior, oh and by a total coincidence their preferred ranking gives them a direct financial benefit. Just a complete failure to imagine the larger issues.
awinter-py 2y ago
thought this was going to be about the 'log in with google' popup
tippytippytango 2y ago
These days I usually only search google for an exact phrase or title I got from chatgpt to check if it hallucinated.
- Tagbert 2y ago
  ... And google ignores your exact phrase and returns something its semantics and ads engine determine is close enough.
4b11b4 2y ago
exa.ai good for finding these sites? but yes, many people will still be googling
acegopher 2y ago
The shepherd.com link is the second one that shows up on Kagi, after a Quora link: https://kagi.com/search?q=best+books+on+Battle+of+Midway https://kagi.com/search?q=best+books+on+Battle+of+Midway
- freediver 2y ago
  Same search for those without Kagi account (using 'share search' option): https://kagi.com/search?q=best+books+on+Battle+of+Midway&r=us&sh=KEpiZ-4mCwT_iT76EM0PZA https://kagi.com/search?q=best+books+on+Battle+of+Midway&r=u...
- rc_mob 2y ago
  I would assume every other kagi user has blacklisted quora already
  - bwb 2y ago
    Ya I have, Quora is so bad now, I used to love them long ago.
  - acegopher 2y ago
    Ohhhh... I didn't know you could blacklist, thanks!
- carom 2y ago
  Top result on duckduckgo: https://duckduckgo.com/?t=ffab&q=best+books+on+the+battle+of+midway&ia=web https://duckduckgo.com/?t=ffab&q=best+books+on+the+battle+of...
- uncharted9 2y ago
  Top result in AI overview and 4th in the general search on Brave Search https://search.brave.com/search?q=best+books+on+the+battle+of+midway https://search.brave.com/search?q=best+books+on+the+battle+o...
twothamendment 2y ago
Only somewhat related, but I was just complaining about search in Gmail. I normally use Thunderbird and didn't have it, so I used the web. Basic searches on the subject were so bad! I even put quotes around it and tried all of my Google-foo. It felt like I was fighting some AI that was sure it knew what I wanted despite my "exact phrase" search.
AyyEye 2y ago
I woke up this morning. Then checked my mailchimp subscriptions and my grow.me subscriptions And my substack subscriptions Made sure my Cloudinary was properly configured And my newrelic analytics And my sentry analytics And my rlcdn analytics And my growplow analytics And my 33across analytics And my Scorecard research analytics And my openxcdn analytics And my trustarc analytics And my creativecdn analytics And my Google Tag Manager And my Google Analytics Then I checked my Mediavine ads And my adsrvr ads And my adform ads And my adnxs ads And my yieldmo ads And my criteo ads And my mediavine ads And my pubmatic ads And my id5-sync ads And my rubicon-project ads And my triplelift ads And my pghub ads And my zemanta ads And my cognitivlabs ads And my doubleverify ads And my media.net ads And my kargo ads And my Amazon ads And my Google Adsense When I finally got to WeWork I booted up my Macbook And checked my Google Workspace email Our Amazon Affiliate account was approved And I checked our private Github issues for tasks Then I let Microsoft CoPilot write Stripe integration I write a new post for my Substack Finally using my Chrome browser and ycombinator's platform I posted: 'Google is killing the independent web'
- rockskon 2y ago
  What does any of what you cited have to do with the independent web? Should he have clarified "properties not backed by tens, hundreds, thousands, or millions of millions of dollars?"
  - AyyEye 2y ago
    I decided to remove my browser condoms and check Shepherd's network traffic. Updated my post to better reflect how "independent" the website is. That's less than half of the malware it tried to run on my computer but I got bored checking all the domains it tried to run code from. It's repetitive. I get it, you put on every tracker and ad network you could find, and put your stuff on a bunch of CDNs. Everything that can be outsourced to big tech was outsourced to big tech. How very independent.
    - bwb 2y ago
      Great question :) Independent means that we are a small team and trying to create something cool for people. It means we are not part of Forbes or some giant publication that is focused purely on money and spamming the shit out of you :). You want a diverse and indie web so that the only search results and voices you here are Forbes, Disney, and other giant media conglomerates. Independent does not mean we don't eat. Independent does not mean we don't have server bills. Independent does not mean our part-time developer works for free. Independent does not mean our designer works for free. There is no malware on the website crazy; it's just an ad network that pays for our servers. I will remove all those ugly display ads as soon as we have enough income from readers and authors directly.
pcdoodle 2y ago
[flagged]
G_o_D 2y ago
Always use google in incogniti so they dont give personalised results, thats the least we can do before google search completely dies, Or just prevent google any cookie permission, keep it session only As far as query trackers on google urls, they are necessary One good case of necessity of google tracking parameters is --> you search for movie or related, but muktiple movies by same name have been made , you dont remember year, you do remember some actor or story description Searxg and google will in show with media carda foe movies you ckick iy and yoi get exact same film with year yoi wantes Wait ->I am telling above because when that mefia image of film i inspecred itsurl it was just Googke ?search=filmname&teackers The search ket wasnt modified with year same as i searched first yet it gave me the desires year film may be because of trackers &ved amd all paramers link tags to search teaukts its loke ctoss maching in databases
Seattle3503 2y ago
The author talks about how long users spend on his page, but can Google track that? Has ubiquitous tracking blocking, and Google's failure to adapt, eroded the quality of search?
- aaomidi 2y ago
  Ubiquitous tracking blocking? Where?
- DrillShopper 2y ago
  We don't even have ubiquitously used ad blocking right now. Do you think some normie is even going to know what a tracker is and why they should block it?
- bwb 2y ago
  There have been links from the DOJ case showing Google is pulling in Chrome data. But its just one data point and nobody knows. It seems more likely at this stage that Google has lost control of their AI system. they trained it on x websites, and indie websites got swept into that. And they don't care.
maytc 2y ago
Switched to Perplexity Pro and haven't looked back
- ashleyn 2y ago
  Ctrl+F'd for Perplexity. I knew Google was cooked the minute Perplexity worked better for questions about an obscure embedded systems SDK. It has little documentation, but a lot of mailing list and github issues. Google spits out the front page of the project and shrugs; Perplexity actually answers the question. The usual caveats for LLM hallucination apply.
- untangle 2y ago
  Same. Try the "books on the Battle of Midway" query on Perplexity. The results are great and include the book mentioned in the article (authored by the Naval Aviator).
- NetOpWibby 2y ago
  I'm already paying for Kagi and Claude. Is Perplexity Pro worth the extra subscription?
neilv 2y ago
Any idea how many Googlers still have the pre-dotcom-boom Internet technologist-citizen mindset? I know a bunch flocked to Google early on. But the landscape has changed a lot since then, and many people weren't even born until after the Internet and society were very different.
- bwb 2y ago
  All my friends at Google left so they could get things done... I worry about that company.
keyle 2y ago
Google got so big it swallowed the internet. It now has digested it and what is left is... this. I used to love crafting websites and cared about SEO. What's the point now, no one is going to find your content. It won't even be on the third page. Google will answer questions by regurgitating whatever it swallowed on your websites and presume no one will click through, it won't even bother marking the authors. Instead it appears to be prioritising whichever website is going to give it revenue first, e.g. the click farms. The regular folks don't care, they google for stuff like "am I dying if I have a pimple?" (to which the answer is always yes, apparently). No one does actual meaningful research using Google anymore, if you do, good luck, get your gloves out <picture of dinosaur poop in Jurassic Park>. The global internet as it stands is close to dead. Discoverability of "cool" things is down to social media, tricked by "influencers", who are tricked by marketing themselves. We need a hard reset button, it needs to start from the ground up with site rings, and good content. Ah... that last part, "good content", is now stuffed with AI Samey McSamey sounding text. I really don't see a way out of here. The funny part is we used to think that the internet was going to change the world. We thought all idiots needed was information. Access to information would fix the world! Instead, it only has given the village idiots a global voice: if you can think of some dumb crazy thing, you'll find dumb crazy people agreeing with you, so you must be right! I've been on the internet since 1997 and I think it's the worst it's ever been.
- djbusby 2y ago
  Opportunity for a new discovery engine. Not Kagi (I buy it) but a search focused on the small web, find stuff not in the top 100k sites.
  - johnnyanmac 2y ago
    I wonder how feasible that is without leveraging one of the two existing search engines in the backend. I always pitted a general search engine as a top 5 difficult tech project to go for.
  - DrillShopper 2y ago
    Enjoy being absolutely crushed by Google and having to fight to get users to even know you exist, let alone use you
g8oz 2y ago
Well they replaced the Yahoo alum accused of tanking search quality with a McKinsey guy....I'm sure things will get better now /s. https://www.wheresyoured.at/requiem-for-raghavan/ https://www.wheresyoured.at/requiem-for-raghavan/
NetOpWibby 2y ago
Why don't people just stop using Google? And by people, I mean everyone here. Whenever this point comes up, I see people claim they ONLY see the results they want in Google. How would you know if you don't actually use anything else? Kagi is excellent search. Neeva was pretty great when it was active. DuckDuckGo is passable. Idk how Qwant gets money but it's been around a bit. Complaining about the same thing forever and expecting a change doesn't make any sense. Y'all are in abusive relationships with Google and refuse to leave. Sure, your job may use Google Suite and you need to make money. What about the rest of your life? Stop hitting yourself.
- johnnyanmac 2y ago
  Once I have stable income I do plan to use Kagi. Though I do recognize that even Kagi partially has Google Search underneath. Even if you fully de-googlefy it's hard to not be indirectly supporting Google. Been working on it for 2 years now and even then I'm more or less locked into 3 services 1. Gmail, 20 year old account I use for pretty much everything business. Even if I move I'll need to forward Gmail stuff to a new email for years. 2. YouTube. Pretty self explanatory (and the go to for why monopolies are always bad). Trying to avoid it entirely is like trying to avoid dang Twitter. Too many other companies use it as a go to for any video, no matter how inefficient (nothing better for graphical showcases than nitrate compression ruining all the details) 3. Play store. Used android all my life and while I can mostly move out I will be missing some critical apps from that result (financial apps are a big example) It's a network effect like any other for 2 of those, and a lock in part of my online identity for another.
  - NetOpWibby 2y ago
    I can't help you with points 1 and 3 but for 2: do you comment on YouTube videos or upload your own? If you just consume content (like I do), you can subscribe to your favorite channels with RSS. Go to a channel and inspect source for "rss" to find the link. I'm partial to NetNewsWire for my feed reader.
- throwaway48476 2y ago
  Because the problem isn't just Google. The internet is a garden whose quality content was left to wither.
squidhunter 2y ago
The indie web was around before google and it will be around long after google is gone. I would argue that the indie web has incurred a much larger loss from people thinking seo/engagement metrics are something worth optimizing. Many of the best examples of the indie/small web don’t have js tracking and little to no css.
GolfPopper 2y ago
"It is difficult to get a man to understand something when his salary depends upon his not understanding it." - Upton Sinclair
nextworddev 2y ago
The real problem is that Google solves for the needs of the lowest common denominator of content consumers, which leads inevitably to enshittification
Havoc 2y ago
Big adtech cares not as long as the ads keep flowing
pixodaros 2y ago
Visits from Google to one of my independent sites have been about the same in 2022, 2023, and 2024. Visits to another have roughly doubled from year to year. I don't use Google Search or surveillance ads but I don't have the issue that they report. AyyEye's observation that their site is loaded with trackers and what look a lot like affiliate marketing links is one reason why Google Search might not like it any more.
- bwb 2y ago
  I am glad you got lucky, I hope it holds up :) Stats are fine, as sometimes people mistake a single data point for a truth. Might take a look at my post as there are links as this is a huge issue affecting many many many websites.
  - pixodaros 2y ago
    I don't have access to your "many friends and acquaintances who are going bankrupt and shutting down fantastic independently run websites." I absolutely don't have access to the internals of their businesses to decide whether I agree that their problem is Google Search being hostile to independent sites. But I do have access to the traffic stats for the sites which I run. Many of the sites with similar complaints seem to focus on product reviews and recommendations and have been hit by a specific Google policy which favors trusted and well-known domains for those types of search, even if that domain is not trusted for product reviews in particular. (So $glossyMagazineInNYC can open a line of product reviews and get lots of Google traffic, while $smallProductReviewBusiness loses traffic).
deleted 2y ago
[deleted]
mikewarot 2y ago
Long ago I wrote a blog post[1] about intermittent hardware, and not letting it suck you in. This was posted in a Blogger/blogspot blog, which is owned by Google. This morning Google couldn't find it, neither could Bing. [1] https://mikewarot.blogspot.com/2007/10/mikes-law-of-intermittent-hardware.html?m=1 https://mikewarot.blogspot.com/2007/10/mikes-law-of-intermit...
- freediver 2y ago
  Kagi can. https://kagi.com/search?q=%22Mike%27s+law+of+intermittent+hardware%22&r=us&sh=bjj7DxaRJxcC0QBm-biFpw https://kagi.com/search?q=%22Mike%27s+law+of+intermittent+ha... (granted, I had to perform search with quotes on, but both Google and Bing still fail) Hopefully demonstrates the difference between legacy and a modern search engine whose entire purpose of existence is to surface what you want to find, not what other people want you to see. (wave from Kagi founder here)
- deleted 2y ago
  [deleted]
Sincere6066 2y ago
is there a way to detect and block substack?
donatj 2y ago
Checking my Search Console Tools recently, the number of "Crawled - currently not indexed" pages across all my sites has risen sharply recently, especially on sites where the age of the content is more than a few years. I put a fair bit of time into trying to improve the sites with JSON-LD and breadcrumbs and what not. It seems to have helped just a little bit. I don't make any money off any of it, but it's still kind of irritating that no one can find it.
AlienRobot 2y ago
As much as I share the frustration, what does Google gain from providing a better service? Its only serious competitor is Bing, which isn't even a search engine anymore but a billboard for Microsoft to advertise ChatGPT.
- bwb 2y ago
  Pride. Happiness. Contentment. A workforce that is motivated and happy. People don't want just money.
throwaway984393 2y ago
[dead]
hhdhdbdb 2y ago
Google can't please all of the sites all of the time, or all the visitors. It is too big to evem worry about that 4k a day clicks for one site. It is like us optimizing the expense of 0.01c. It makes a difference when that 0.01c is an API call that you call a million times. But it only surfaces if you do aggregate it. Therefore this problem can only even be seem by Google if it can be surfaced in aggregate overy say a billion queries. I wonder how that can be done. Probably only can be done using data. Which means spying on people in various ways. And making assumptions about length of time on site equals quality. They probably use machine learning too. There may be no reason for the lost rankings other than a wind change caused by some updated parameters in an OKR chasing model.
- hn_throwaway_99 2y ago
  I realize "did you actually read the article" is against HN guidelines, but what else am I supposed to say when I see this at the top of the comment thread? I mean when you say "Google can't please all of the sites all of the time, or all the visitors.", I wholeheartedly agree, but this blog post was excellently sourced with data that shows exactly how Google raising sites that any reasonable human would say are considerably shittier than this site that is getting down ranked. It also seems pretty clear that the things that have changed are Google's ranking algorithm at specific points. > They probably use machine learning too. There may be no reason for the lost rankings other than a wind change caused by some updated parameters in an OKR chasing model. That is literally what TFA says in the very first section: "Some people believe they have lost control of their AI ranking systems, ..."
  - hhdhdbdb 2y ago
    I think I said a lot more than TFA on those points.
paulista_tcb 2y ago
you should check out exa.ai/search if you miss indie content!
cperez081890 2y ago
[dead]
amluto 2y ago
One thing I find especially bizarre about the current situation: the pages that are making it to the top aren’t even pretending to be real content. If they were difficult-to-detect LLM-generated pages, that would be one thing. But they’re generally extremely low-effort affiliate spam, mostly claiming that they researched something “so I don’t have to”, followed by a bunch of Amazon links and explicitly acknowledged scraped reviews, and finally an obviously uninformative summary. They don’t even pretend to have real content! What is Google doing?
- bwb 2y ago
  Ya, that is what is frustrating here. A lot of us believe that Google trained their AI on spam websites, but in the process, the AI now identifies a lot of things that independent websites have on them versus big brands like Forbes, which spam and license out huge sections of their websites to spammers. So we got pulled into this and Google doesn't care about the false positives. At this point, a lot of websites are not coming back. So get used to Forbes and other large brands being the only results along with AI bots on Quora.
StressedDev 2y ago
Google recently changed who is running search. Basically, Prabhakar Raghavan was moved from running Search and ads to being Chief Technologist. Nick Fox is the new person running Google Search. This happened on 10/17/2024. For more information, please see https://searchengineland.com/google-shakes-up-leadership-raghavan-becomes-chief-technologist-447638 https://searchengineland.com/google-shakes-up-leadership-rag... and https://www.theverge.com/2024/10/17/24272786/google-search-prabhakar-raghavan-nick-fox https://www.theverge.com/2024/10/17/24272786/google-search-p... . I wonder if Prabhakar Raghavan was moved out of Google Search because of Search's decline in quality?
bun_terminator 2y ago
Hi stranger, please stop messing with the scrollbar
renegat0x0 2y ago
1) It has already been proven by research that Google quality of results slips from year to year [1] 2) Google has many incentives to make the search more difficult for you 3) Google has proven that it prefers money over quality of results with allowing "malvertising" 4) It is true that the landscape is more difficult. There are more walled gardens, to which even Google might not have access. There are more scams, casinos. More AI slop. The game was always hard on the other hand, so these are just 'excuses' 5) Why so often I see in Google results leading to major news sites instead of normal links? 6) If I write "Warhammer" I would expect thousands and thousands of pages in results. I think that Google prefers "content" over "quality". "newer" is "better". I would expect thousands of fan pages, which do exists, but are not crawled, or forgotten. Why can't I browse older pages? Why is there a limit to 10 pages? 7) For "Emulation" first page leads to "wikipedia", "cambridge dictionary", "vocabulary", it is so f boring [1] https://www.404media.co/google-search-really-has-gotten-worse-researchers-find https://www.404media.co/google-search-really-has-gotten-wors...
MrSkelter 2y ago
This problem is age old and always the same. Optimizing for Google is a fools errand. You can win in the short term, but when their priorities change you will find your website excluded for the very things that boosted it previously. Google are unreliable and untrustworthy. Their focus is ad revenue for themselves and nothing else. Build sites for humans and let Google do what it wants to.
znpy 2y ago
Maybe we should just start looking at other search engines seriously and without irony. How’s shepherd.com (for example) doing on bing or duckduckgo? Maybe it’s time for google to die.
tessierashpool9 2y ago
but to google you're just a desperate flea from the indie web.
p3rls 2y ago
Great work as always Danny Sullivan!
fsckboy 2y ago
>Explore only the books that readers love. that doesn't sound like "the independent web", that sounds like an airport bookseller.
tammybullarb 2y ago
Take the same time and go back in and see what the other people have said about how they get a job to make a good night and see how you want it out land on a good work! I'm sure if poor is going on that e and the next year Prime Minister of course he would. And the other one of my own parents were that he had been working with him and he didn't want the money he was. But the next thing was the other people that had to make the other people that had no idea how to get out. But the only thing I know about this would have Saints of the game

256 points by bwb 2y ago

If you're still barking in the general direction of google and similarly sized search engines, good luck. I'd rather look towards kagi and brave.

viraptor 2y ago
I feel too that we're already past the point where they could fix things. They must be aware of issues and do the exact opposite of fixing them. It's a question how fast we can convince general public to move away, rather than when Google will improve.
- jasonvorhe 2y ago
  I wouldn't give much regarding the general public. If you have a small blog they're usually not your target audience. Accept that the days of "making a few bucks off of advertising" on hobby publishing are over, focus on writing about things you care about without it reeking of being written by ChatGPT and chances are people will come at some point. If small publishers don't change their behavior in radical ways, nothing will ever change.
  - viraptor 2y ago
    > nothing will ever change. That's probably what MySpace people thought. And Friendster. And Altavista. And ...
    - jasonvorhe 2y ago
      I don't get what you're trying to say. I'm not arguing in favor of doing nothing in an ever-changing market, which is what Myspace and Friendster did most of the time. I'm saying the old world of "have good content/SEO and you'll see some traction thanks to Google" is dying and that you can either adapt or drown in a flood of LLM-created SEO blackhat nonsense. Google does not care about you and they've been in the "too big to fail" corner for too long. If you want content made by humans to continue to exist and want to see some of them be able to make a buck off of it, support smaller search engines and creative people and their publications.
      - viraptor 2y ago
        I mean that services will come and go. Some will stay for longer than others, but even without a coordinated action, Google may go away. It's also losing to Perplexity and ChatGPT with many people. And many geeks are very vocal about their love for Kagi. (Including me) Things may change over time and may not be too big to fail.
wbl 2y ago
And how is the experience? Even with Bing I find I have to switch to Google to get useful results for my queries.
- senko 2y ago
  Kagi user here, I almost never need to !g. Used DDG a few years back and stopped after I had to fallback to Google for most qs.
- girvo 2y ago
  Google is so odd these days, its like there are two "streams" of results. One is the kind of useless slop that the article in question is rightfully complaining about, but other results are still useful and high quality. The second stream is what I seem to get when I'm forced to "!g" when I'm using DuckDuckGo, and I can't work out why theres this stratification, this separation. Years ago, it was very rare to get crap results from Google. Now its far more common.
- alpaca128 2y ago
  It took me about two weeks until I got consistently good results with DDG, it often just takes about one more search term because it doesn't try to (often incorrectly) guess the context based on search history like Google does. That was years ago, since then DDG has gotten better and Google a lot worse. I could understand your complaint if it was about image search, but I don't see how anyone can look at today's Google results and think they're still superior.
- viraptor 2y ago
  Kagi is simply better in every way I can think of and I've been a paying user for over a year now.
- justinclift 2y ago
  Haven't needed to use Google at all since using Kagi several months ago, whereas when using Duck Duck Go for a few years (prior to Kagi) I'd (very occasionally) need to.
lolinder 2y ago
Kagi is a viable choice for individual people, but it's not an answer for the concern of how to get traffic to visit your site and click on your affiliate links (which is what TFA is about). Kagi works well for users precisely because it's small enough that no one benefits from gaming its algorithms.
- rockskon 2y ago
  How is Kagi faring after Google's exclusivity deal with Reddit?
  - lolinder 2y ago
    Um, I don't know when that would have kicked in because I haven't noticed a difference. Reddit still shows up whenever it's relevant, and the Forums lens turns up even more Reddit if you're worried you're missing any on the initial search.
  - klooney 2y ago
    I think Kagi uses Google's APIs as a part of their mix of search results.
- Ferret7446 2y ago
  Why is it Google's job, who TFA isn't paying AFAIK, to get traffic to visit their site?
  - lolinder 2y ago
    I don't think it is, but recommending Kagi as a solution for their complaint misses their point.

jqpabc123 2y ago

Simple answer --- stop using Google. Get your friends and family and other internet users to do the same. The only thing likely to get their attention is if enough people follow suit.

23B1 2y ago
"Simple answer, just get everyone to stay home for three weeks and the virus will die out" "Simple answer, just stop electing corrupt politicans" "Simple answer, just stop war" k
- deleted 2y ago
  [deleted]
- senko 2y ago
  Subscribing to Kagi is definitely easier (and less costly!) than handling the pandemic, ensuring world peace and utopian government. Edit to add: and since we're talking about indie web: https://search.marginalia.nu/ https://search.marginalia.nu/
  - nemomarx 2y ago
    Me subscribing to it is. Getting everyone I know to subscribe to it is at least a step more complex than getting people to move to Firefox, and I haven't quite worked out a strategy for that yet.
    - justinclift 2y ago
      Search results being absolutely shit is probably more of a problem for people than their choice of browser. When someone complains to you directly about Google search results being shit, maybe point them to Kagi and mention it has a free trial account (100 searches still?) that's not time limited. That way they'll have a solution they can turn to the next time Google pisses them off significantly. :)
      - 2y ago

fny 2y ago

I find it ironic that there’s a desperate plea for the indie web from a Substack blog.

croes 2y ago
To get more visibility?
- CSSer 2y ago
  I think the irony works either way.
- CharlieDigital 2y ago
  If OP was going for visibility by posting it HN, Reddit, etc. then does it matter if it was Substack or his own blog or even the Shepherd site?
  - johnnyanmac 2y ago
    Technically, yes. If the thesis is "Google search is inaccurate and overrun with SEO optimized slop and ads", them using a blog utilizing that same SEO optimization feels a bit ironic. Feeding the beast you want to change.
    - bwb 2y ago
      I don't quite understand, I still want Google traffic, they have a monopoly. Nothing has changed there.
- bwb 2y ago
  That is a private blog, so I don't care if anyone sees it. I just wanted something I updated every 2 to 3 weeks with my frustrations, wins, and so on as I build Shepherd.com. I was on Twitter's free email thing but they shut down and imported to Substack. I just didn't want to pay for anything as the only person who reads it is me and my mom. It has 100 readers on it, as its just for me really.
2y ago

deleted 2y ago

[deleted]

grugagag 2y ago

Google didn’t shit the bed, they just did whatever was profitable to them and they’ll continue doing so until they either fail or there’s no more profit to be squeezed. They dropped the “don’t be evil” motto a long tome ago.

skybrian 2y ago

It looks like shepherd.com is a book review site that doesn't have any reviews, just ratings. It links to Amazon. Is this really the sort of content Google should be returning? Edit: it seems I missed the link to the actual book reviews because the link text is uninformative: "Chosen by 1 person - see why." (Sometimes it goes to reviews, sometimes not.) And the word "review" never appears on the pages that have reviews. Seems like bad SEO? If you're looking for book reviews, here's a website with some pretty great reviews: https://www.thepsmiths.com/ https://www.thepsmiths.com/ (Content warning: the authors are conservative.)

passwordoops 2y ago
Just went over there and yes they have reviews from readers https://shepherd.com/book/exit-strategy https://shepherd.com/book/exit-strategy
- skybrian 2y ago
  Thanks, I missed that.
bwb 2y ago
Hi Brian, we don't have reviews at this stage; we have purely positive recommendations. And we push readers toward a very specific format for why the book resonated with them (Gestalt psychology so you see it through their eyes). Given the nightmare that is Goodread's unmoderated review sludge, I wanted to focus on the positivity and why someone loved a book, so we are uniquely focused on highlighting only books that readers love. In 2025 I will be adding "reviews" of any books using our Book DNA review format (which tries to narrow in on why someone liked/disliked a book so we can match you with people who share your Book DNA). A big part of my mission is to focus only on books that people love and positivity.
willywanker 2y ago
Was that content warning really necessary, or do we also tend to use them for liberal websites (for whatever definition of liberal) ?

23B1 2y ago

There are people inside Google who can put a stop to this, if they merely stand up and say "this is wrong" including their CEO. It simply requires bravery and faith in Google's institutional ability to continue to profit doing different things. Big companies have pivoted before on the heels of brave executives. This applies as much to search as it does to privacy, AI, their transparency, their support of open source, and their weaponization of their browser. It's a crying shame to see how enshittified a company that could be changing the world has become.

immibis 2y ago
And a willingness to be fired the next day. Even the CEO can be fired.
- mcmcmc 2y ago
  So it’s the shareholders and the board who are at fault then. I’d posit that anyone holding Google stock is part of the problem and responsible for their continued pattern of anti-user and anti-consumer behavior.
  - DrillShopper 2y ago
    It's actually late stage capitalism that is at fault. Line must always be going up, and if it doesn't go up as fast as someone thought it would then we make it go down. If line goes sideways we sell and make it go down. It must go up. Always up. Or it will go down.
    - mcmcmc 2y ago
      You can’t blame the system and pretend none of that lands at the feet of people willingly engaging with it for their own benefit.
      - DrillShopper 2y ago
        Some of it absolutely lands at the feet of the people willingly engaging with it, but there are far, far more people who are caught up in and and only go along with it to have a roof over their head. I don't think the burger flipper at McDonalds, the stocker at Walmart, the barista at Starbucks, the bartender at the bar down the street, the person working the register at the gas station, or the picker at Amazon (just to name a few) share in this blame. They're all trying to eke out an existence for themselves and their families.

immibis 2y ago

Hi (company that benefits from hurting me), please stop hurting me. Company: No.

niobe 2y ago

8-10 years google search was amazing. A well crafted query would hit informational gold most of the time. I've been noticing and commenting on the decline privately for most of the period, but it's only in the oast year this seems to have come to broader awareness. There's an argument that in an information economy, searching for information should be treated like a public utility necessary for the functioning of society. I'm not making that argument but when you experience the long slippery slope of degradation of a service that was near ideal for the technology of the time, it does xome to mind. That's the thing. It DID work. Really well for a while. But it was always atomic and context-less. We now have the opportunity to make it even better by refining results through dialogue. I hope someone does.. soon.

zx8080 2y ago
> We now have the opportunity to make it even better by refining results through dialogue. I hope someone does.. soon. TBH, I don't want dialog (too long and slow, leave dialogs for human to human comms), I just need my query words to exist in the result pages. Nothing more. Google messed this up long time ago.
- throwaway48476 2y ago
  I wouldn't mind a dialog that helps me build an advanced query.
  - nradov 2y ago
    Have you tried advanced search? https://www.google.com/advanced_search https://www.google.com/advanced_search
    - throwaway48476 2y ago
      I'm aware of advanced search. I'd like it to be more user friendly and widely used. I think a chat bot would help.
- munificent 2y ago

danpalmer 2y ago

I have no specific information here, never seen/used Shepherd until I saw a list on there a few days ago. Disclaimer, I work for Google, but not on anything related to this space, and this is based on my previous job where we did some SEO for an ecommerce site. The example list given just looks a lot like spam when you squint. It's a list of affiliate links to buy products, and there are many HN threads talking about the abundance of affiliate link aggregators being a blight on the web. The commentary does look useful, but distinguishing between good commentary and bad commentary is hard, whereas distinguishing between a site designed to extract affiliate commission vs one more about the content is easier. The comparison given to the other results here is frustrating, I know, but probably not a valid experiment. All the major search engines change results based on the user using them, or the IP address, or the region, or whatever, so it's impossible to know what others see. The developer of a book-focused shopping site is likely to get very skewed results for a book related query. My results were noticeably better. The author says that a Bookshop.org list they created that links back to Shepherd is ranking #2, and this kinda makes sense to me. Bookshop.org sells the books, it makes sense that would rank above a site that only links to (and makes money from) sites that sell books. SEO, and people getting annoyed at not ranking, has been a thing for 25+ years, I don't think this instance is any different.

smallerize 2y ago
(That's not what dearth means.)
- Jerrrrrrry 2y ago
  portmanteau of "dithering" and "blight" and "earth"
- danpalmer 2y ago
  Whoops, thanks.
strunz 2y ago
Do you use Google anymore? It has gotten so bad, I regularly can't find things on it anymore, no matter how I search for things it will keep showing the same results. You seem to have blinders on. How do you explain the site getting less and less traffic over time?

jeffbee 2y ago

I've had my lifetime quota of hearing the rants of guys who think their search quality metric is objectively superior, oh and by a total coincidence their preferred ranking gives them a direct financial benefit. Just a complete failure to imagine the larger issues.

awinter-py 2y ago

thought this was going to be about the 'log in with google' popup

tippytippytango 2y ago

These days I usually only search google for an exact phrase or title I got from chatgpt to check if it hallucinated.

Tagbert 2y ago
... And google ignores your exact phrase and returns something its semantics and ads engine determine is close enough.

4b11b4 2y ago

exa.ai good for finding these sites? but yes, many people will still be googling

acegopher 2y ago

The shepherd.com link is the second one that shows up on Kagi, after a Quora link: https://kagi.com/search?q=best+books+on+Battle+of+Midway https://kagi.com/search?q=best+books+on+Battle+of+Midway

freediver 2y ago
Same search for those without Kagi account (using 'share search' option): https://kagi.com/search?q=best+books+on+Battle+of+Midway&r=us&sh=KEpiZ-4mCwT_iT76EM0PZA https://kagi.com/search?q=best+books+on+Battle+of+Midway&r=u...
rc_mob 2y ago
I would assume every other kagi user has blacklisted quora already
- bwb 2y ago
  Ya I have, Quora is so bad now, I used to love them long ago.
- acegopher 2y ago
  Ohhhh... I didn't know you could blacklist, thanks!
carom 2y ago
Top result on duckduckgo: https://duckduckgo.com/?t=ffab&q=best+books+on+the+battle+of+midway&ia=web https://duckduckgo.com/?t=ffab&q=best+books+on+the+battle+of...
uncharted9 2y ago
Top result in AI overview and 4th in the general search on Brave Search https://search.brave.com/search?q=best+books+on+the+battle+of+midway https://search.brave.com/search?q=best+books+on+the+battle+o...

twothamendment 2y ago

Only somewhat related, but I was just complaining about search in Gmail. I normally use Thunderbird and didn't have it, so I used the web. Basic searches on the subject were so bad! I even put quotes around it and tried all of my Google-foo. It felt like I was fighting some AI that was sure it knew what I wanted despite my "exact phrase" search.

AyyEye 2y ago

I woke up this morning. Then checked my mailchimp subscriptions and my grow.me subscriptions And my substack subscriptions Made sure my Cloudinary was properly configured And my newrelic analytics And my sentry analytics And my rlcdn analytics And my growplow analytics And my 33across analytics And my Scorecard research analytics And my openxcdn analytics And my trustarc analytics And my creativecdn analytics And my Google Tag Manager And my Google Analytics Then I checked my Mediavine ads And my adsrvr ads And my adform ads And my adnxs ads And my yieldmo ads And my criteo ads And my mediavine ads And my pubmatic ads And my id5-sync ads And my rubicon-project ads And my triplelift ads And my pghub ads And my zemanta ads And my cognitivlabs ads And my doubleverify ads And my media.net ads And my kargo ads And my Amazon ads And my Google Adsense When I finally got to WeWork I booted up my Macbook And checked my Google Workspace email Our Amazon Affiliate account was approved And I checked our private Github issues for tasks Then I let Microsoft CoPilot write Stripe integration I write a new post for my Substack Finally using my Chrome browser and ycombinator's platform I posted: 'Google is killing the independent web'

rockskon 2y ago
What does any of what you cited have to do with the independent web? Should he have clarified "properties not backed by tens, hundreds, thousands, or millions of millions of dollars?"
- AyyEye 2y ago
  I decided to remove my browser condoms and check Shepherd's network traffic. Updated my post to better reflect how "independent" the website is. That's less than half of the malware it tried to run on my computer but I got bored checking all the domains it tried to run code from. It's repetitive. I get it, you put on every tracker and ad network you could find, and put your stuff on a bunch of CDNs. Everything that can be outsourced to big tech was outsourced to big tech. How very independent.
  - bwb 2y ago
    Great question :) Independent means that we are a small team and trying to create something cool for people. It means we are not part of Forbes or some giant publication that is focused purely on money and spamming the shit out of you :). You want a diverse and indie web so that the only search results and voices you here are Forbes, Disney, and other giant media conglomerates. Independent does not mean we don't eat. Independent does not mean we don't have server bills. Independent does not mean our part-time developer works for free. Independent does not mean our designer works for free. There is no malware on the website crazy; it's just an ad network that pays for our servers. I will remove all those ugly display ads as soon as we have enough income from readers and authors directly.

pcdoodle 2y ago

[flagged]

G_o_D 2y ago

Always use google in incogniti so they dont give personalised results, thats the least we can do before google search completely dies, Or just prevent google any cookie permission, keep it session only As far as query trackers on google urls, they are necessary One good case of necessity of google tracking parameters is --> you search for movie or related, but muktiple movies by same name have been made , you dont remember year, you do remember some actor or story description Searxg and google will in show with media carda foe movies you ckick iy and yoi get exact same film with year yoi wantes Wait ->I am telling above because when that mefia image of film i inspecred itsurl it was just Googke ?search=filmname&teackers The search ket wasnt modified with year same as i searched first yet it gave me the desires year film may be because of trackers &ved amd all paramers link tags to search teaukts its loke ctoss maching in databases

Seattle3503 2y ago

The author talks about how long users spend on his page, but can Google track that? Has ubiquitous tracking blocking, and Google's failure to adapt, eroded the quality of search?

aaomidi 2y ago
Ubiquitous tracking blocking? Where?
DrillShopper 2y ago
We don't even have ubiquitously used ad blocking right now. Do you think some normie is even going to know what a tracker is and why they should block it?
bwb 2y ago
There have been links from the DOJ case showing Google is pulling in Chrome data. But its just one data point and nobody knows. It seems more likely at this stage that Google has lost control of their AI system. they trained it on x websites, and indie websites got swept into that. And they don't care.

maytc 2y ago

Switched to Perplexity Pro and haven't looked back

ashleyn 2y ago
Ctrl+F'd for Perplexity. I knew Google was cooked the minute Perplexity worked better for questions about an obscure embedded systems SDK. It has little documentation, but a lot of mailing list and github issues. Google spits out the front page of the project and shrugs; Perplexity actually answers the question. The usual caveats for LLM hallucination apply.
untangle 2y ago
Same. Try the "books on the Battle of Midway" query on Perplexity. The results are great and include the book mentioned in the article (authored by the Naval Aviator).
NetOpWibby 2y ago
I'm already paying for Kagi and Claude. Is Perplexity Pro worth the extra subscription?

neilv 2y ago

Any idea how many Googlers still have the pre-dotcom-boom Internet technologist-citizen mindset? I know a bunch flocked to Google early on. But the landscape has changed a lot since then, and many people weren't even born until after the Internet and society were very different.

bwb 2y ago
All my friends at Google left so they could get things done... I worry about that company.

keyle 2y ago

Google got so big it swallowed the internet. It now has digested it and what is left is... this. I used to love crafting websites and cared about SEO. What's the point now, no one is going to find your content. It won't even be on the third page. Google will answer questions by regurgitating whatever it swallowed on your websites and presume no one will click through, it won't even bother marking the authors. Instead it appears to be prioritising whichever website is going to give it revenue first, e.g. the click farms. The regular folks don't care, they google for stuff like "am I dying if I have a pimple?" (to which the answer is always yes, apparently). No one does actual meaningful research using Google anymore, if you do, good luck, get your gloves out <picture of dinosaur poop in Jurassic Park>. The global internet as it stands is close to dead. Discoverability of "cool" things is down to social media, tricked by "influencers", who are tricked by marketing themselves. We need a hard reset button, it needs to start from the ground up with site rings, and good content. Ah... that last part, "good content", is now stuffed with AI Samey McSamey sounding text. I really don't see a way out of here. The funny part is we used to think that the internet was going to change the world. We thought all idiots needed was information. Access to information would fix the world! Instead, it only has given the village idiots a global voice: if you can think of some dumb crazy thing, you'll find dumb crazy people agreeing with you, so you must be right! I've been on the internet since 1997 and I think it's the worst it's ever been.

djbusby 2y ago
Opportunity for a new discovery engine. Not Kagi (I buy it) but a search focused on the small web, find stuff not in the top 100k sites.
- johnnyanmac 2y ago
  I wonder how feasible that is without leveraging one of the two existing search engines in the backend. I always pitted a general search engine as a top 5 difficult tech project to go for.
- DrillShopper 2y ago
  Enjoy being absolutely crushed by Google and having to fight to get users to even know you exist, let alone use you

g8oz 2y ago

Well they replaced the Yahoo alum accused of tanking search quality with a McKinsey guy....I'm sure things will get better now /s. https://www.wheresyoured.at/requiem-for-raghavan/ https://www.wheresyoured.at/requiem-for-raghavan/

NetOpWibby 2y ago

Why don't people just stop using Google? And by people, I mean everyone here. Whenever this point comes up, I see people claim they ONLY see the results they want in Google. How would you know if you don't actually use anything else? Kagi is excellent search. Neeva was pretty great when it was active. DuckDuckGo is passable. Idk how Qwant gets money but it's been around a bit. Complaining about the same thing forever and expecting a change doesn't make any sense. Y'all are in abusive relationships with Google and refuse to leave. Sure, your job may use Google Suite and you need to make money. What about the rest of your life? Stop hitting yourself.

johnnyanmac 2y ago
Once I have stable income I do plan to use Kagi. Though I do recognize that even Kagi partially has Google Search underneath. Even if you fully de-googlefy it's hard to not be indirectly supporting Google. Been working on it for 2 years now and even then I'm more or less locked into 3 services 1. Gmail, 20 year old account I use for pretty much everything business. Even if I move I'll need to forward Gmail stuff to a new email for years. 2. YouTube. Pretty self explanatory (and the go to for why monopolies are always bad). Trying to avoid it entirely is like trying to avoid dang Twitter. Too many other companies use it as a go to for any video, no matter how inefficient (nothing better for graphical showcases than nitrate compression ruining all the details) 3. Play store. Used android all my life and while I can mostly move out I will be missing some critical apps from that result (financial apps are a big example) It's a network effect like any other for 2 of those, and a lock in part of my online identity for another.
- NetOpWibby 2y ago
  I can't help you with points 1 and 3 but for 2: do you comment on YouTube videos or upload your own? If you just consume content (like I do), you can subscribe to your favorite channels with RSS. Go to a channel and inspect source for "rss" to find the link. I'm partial to NetNewsWire for my feed reader.
throwaway48476 2y ago
Because the problem isn't just Google. The internet is a garden whose quality content was left to wither.

squidhunter 2y ago

The indie web was around before google and it will be around long after google is gone. I would argue that the indie web has incurred a much larger loss from people thinking seo/engagement metrics are something worth optimizing. Many of the best examples of the indie/small web don’t have js tracking and little to no css.

GolfPopper 2y ago

"It is difficult to get a man to understand something when his salary depends upon his not understanding it." - Upton Sinclair

nextworddev 2y ago

The real problem is that Google solves for the needs of the lowest common denominator of content consumers, which leads inevitably to enshittification

Havoc 2y ago

Big adtech cares not as long as the ads keep flowing

pixodaros 2y ago

Visits from Google to one of my independent sites have been about the same in 2022, 2023, and 2024. Visits to another have roughly doubled from year to year. I don't use Google Search or surveillance ads but I don't have the issue that they report. AyyEye's observation that their site is loaded with trackers and what look a lot like affiliate marketing links is one reason why Google Search might not like it any more.

bwb 2y ago
I am glad you got lucky, I hope it holds up :) Stats are fine, as sometimes people mistake a single data point for a truth. Might take a look at my post as there are links as this is a huge issue affecting many many many websites.
- pixodaros 2y ago
  I don't have access to your "many friends and acquaintances who are going bankrupt and shutting down fantastic independently run websites." I absolutely don't have access to the internals of their businesses to decide whether I agree that their problem is Google Search being hostile to independent sites. But I do have access to the traffic stats for the sites which I run. Many of the sites with similar complaints seem to focus on product reviews and recommendations and have been hit by a specific Google policy which favors trusted and well-known domains for those types of search, even if that domain is not trusted for product reviews in particular. (So $glossyMagazineInNYC can open a line of product reviews and get lots of Google traffic, while $smallProductReviewBusiness loses traffic).

deleted 2y ago

[deleted]

mikewarot 2y ago

Long ago I wrote a blog post[1] about intermittent hardware, and not letting it suck you in. This was posted in a Blogger/blogspot blog, which is owned by Google. This morning Google couldn't find it, neither could Bing. [1] https://mikewarot.blogspot.com/2007/10/mikes-law-of-intermittent-hardware.html?m=1 https://mikewarot.blogspot.com/2007/10/mikes-law-of-intermit...

freediver 2y ago
Kagi can. https://kagi.com/search?q=%22Mike%27s+law+of+intermittent+hardware%22&r=us&sh=bjj7DxaRJxcC0QBm-biFpw https://kagi.com/search?q=%22Mike%27s+law+of+intermittent+ha... (granted, I had to perform search with quotes on, but both Google and Bing still fail) Hopefully demonstrates the difference between legacy and a modern search engine whose entire purpose of existence is to surface what you want to find, not what other people want you to see. (wave from Kagi founder here)
deleted 2y ago
[deleted]

Sincere6066 2y ago

is there a way to detect and block substack?

donatj 2y ago

Checking my Search Console Tools recently, the number of "Crawled - currently not indexed" pages across all my sites has risen sharply recently, especially on sites where the age of the content is more than a few years. I put a fair bit of time into trying to improve the sites with JSON-LD and breadcrumbs and what not. It seems to have helped just a little bit. I don't make any money off any of it, but it's still kind of irritating that no one can find it.

AlienRobot 2y ago

As much as I share the frustration, what does Google gain from providing a better service? Its only serious competitor is Bing, which isn't even a search engine anymore but a billboard for Microsoft to advertise ChatGPT.

bwb 2y ago
Pride. Happiness. Contentment. A workforce that is motivated and happy. People don't want just money.

throwaway984393 2y ago

[dead]

hhdhdbdb 2y ago

Google can't please all of the sites all of the time, or all the visitors. It is too big to evem worry about that 4k a day clicks for one site. It is like us optimizing the expense of 0.01c. It makes a difference when that 0.01c is an API call that you call a million times. But it only surfaces if you do aggregate it. Therefore this problem can only even be seem by Google if it can be surfaced in aggregate overy say a billion queries. I wonder how that can be done. Probably only can be done using data. Which means spying on people in various ways. And making assumptions about length of time on site equals quality. They probably use machine learning too. There may be no reason for the lost rankings other than a wind change caused by some updated parameters in an OKR chasing model.

hn_throwaway_99 2y ago
I realize "did you actually read the article" is against HN guidelines, but what else am I supposed to say when I see this at the top of the comment thread? I mean when you say "Google can't please all of the sites all of the time, or all the visitors.", I wholeheartedly agree, but this blog post was excellently sourced with data that shows exactly how Google raising sites that any reasonable human would say are considerably shittier than this site that is getting down ranked. It also seems pretty clear that the things that have changed are Google's ranking algorithm at specific points. > They probably use machine learning too. There may be no reason for the lost rankings other than a wind change caused by some updated parameters in an OKR chasing model. That is literally what TFA says in the very first section: "Some people believe they have lost control of their AI ranking systems, ..."
- hhdhdbdb 2y ago
  I think I said a lot more than TFA on those points.

paulista_tcb 2y ago

you should check out exa.ai/search if you miss indie content!

cperez081890 2y ago

[dead]

amluto 2y ago

One thing I find especially bizarre about the current situation: the pages that are making it to the top aren’t even pretending to be real content. If they were difficult-to-detect LLM-generated pages, that would be one thing. But they’re generally extremely low-effort affiliate spam, mostly claiming that they researched something “so I don’t have to”, followed by a bunch of Amazon links and explicitly acknowledged scraped reviews, and finally an obviously uninformative summary. They don’t even pretend to have real content! What is Google doing?

bwb 2y ago
Ya, that is what is frustrating here. A lot of us believe that Google trained their AI on spam websites, but in the process, the AI now identifies a lot of things that independent websites have on them versus big brands like Forbes, which spam and license out huge sections of their websites to spammers. So we got pulled into this and Google doesn't care about the false positives. At this point, a lot of websites are not coming back. So get used to Forbes and other large brands being the only results along with AI bots on Quora.

StressedDev 2y ago

Google recently changed who is running search. Basically, Prabhakar Raghavan was moved from running Search and ads to being Chief Technologist. Nick Fox is the new person running Google Search. This happened on 10/17/2024. For more information, please see https://searchengineland.com/google-shakes-up-leadership-raghavan-becomes-chief-technologist-447638 https://searchengineland.com/google-shakes-up-leadership-rag... and https://www.theverge.com/2024/10/17/24272786/google-search-prabhakar-raghavan-nick-fox https://www.theverge.com/2024/10/17/24272786/google-search-p... . I wonder if Prabhakar Raghavan was moved out of Google Search because of Search's decline in quality?

bun_terminator 2y ago

Hi stranger, please stop messing with the scrollbar

renegat0x0 2y ago

1) It has already been proven by research that Google quality of results slips from year to year [1] 2) Google has many incentives to make the search more difficult for you 3) Google has proven that it prefers money over quality of results with allowing "malvertising" 4) It is true that the landscape is more difficult. There are more walled gardens, to which even Google might not have access. There are more scams, casinos. More AI slop. The game was always hard on the other hand, so these are just 'excuses' 5) Why so often I see in Google results leading to major news sites instead of normal links? 6) If I write "Warhammer" I would expect thousands and thousands of pages in results. I think that Google prefers "content" over "quality". "newer" is "better". I would expect thousands of fan pages, which do exists, but are not crawled, or forgotten. Why can't I browse older pages? Why is there a limit to 10 pages? 7) For "Emulation" first page leads to "wikipedia", "cambridge dictionary", "vocabulary", it is so f boring [1] https://www.404media.co/google-search-really-has-gotten-worse-researchers-find https://www.404media.co/google-search-really-has-gotten-wors...

MrSkelter 2y ago

This problem is age old and always the same. Optimizing for Google is a fools errand. You can win in the short term, but when their priorities change you will find your website excluded for the very things that boosted it previously. Google are unreliable and untrustworthy. Their focus is ad revenue for themselves and nothing else. Build sites for humans and let Google do what it wants to.

znpy 2y ago

Maybe we should just start looking at other search engines seriously and without irony. How’s shepherd.com (for example) doing on bing or duckduckgo? Maybe it’s time for google to die.

tessierashpool9 2y ago

but to google you're just a desperate flea from the indie web.

p3rls 2y ago

Great work as always Danny Sullivan!

fsckboy 2y ago

>Explore only the books that readers love. that doesn't sound like "the independent web", that sounds like an airport bookseller.

tammybullarb 2y ago

Take the same time and go back in and see what the other people have said about how they get a job to make a good night and see how you want it out land on a good work! I'm sure if poor is going on that e and the next year Prime Minister of course he would. And the other one of my own parents were that he had been working with him and he didn't want the money he was. But the next thing was the other people that had to make the other people that had no idea how to get out. But the only thing I know about this would have Saints of the game