55 ms·

Google no longer automatically indexes websites?

599 points by c5karl 3y ago

aabbcc1241 3y ago
An alternative web search architecture, is collaborative sharing the site we made or visit. I made a self hosted spider that only index sites that I've visited and search engine with "and" "or" logics. Thinking to make it a federated search engine by allow each user whitelist domains that are not sensitive to search with the peer. And users can follow/block other users explicitly to avoid spam. https://github.com/beenotung/personal-search-engine https://github.com/beenotung/personal-search-engine
FinnKuhn 3y ago
I tried this an I noticed that Bing doesn't seem to have this problem and has quite a few more pages indexed that Google.
KomoD 3y ago
Site is dead?
- yabones 3y ago
  Snapshot from a couple days ago: https://web.archive.org/web/20230531062024/https://natehoffelder.com/blog/google-no-longer-automatically-indexes-websites-wtf/ https://web.archive.org/web/20230531062024/https://natehoffe...
  - Cockbrand 3y ago
    Or from the other popular archive site: https://archive.is/8bv2T https://archive.is/8bv2T
- jmkni 3y ago
  I thought you were referring to Google for a second lol
- LorenDB 3y ago
  Just the HN Hug O' Death at work again :)
BiteCode_dev 3y ago
I also noticed it takes way more time to index sites. When I moved bitecode.substack.com to https://bitecode.dev https://bitecode.dev, I submitted the sitemap to google and was surprised that it took almost a week to even show a single page as indexed. This used to be a couple of hours to a coupled of days. So something definitely changed, but as usual we can only speculate until a googler come in this thread.
- gsatic 3y ago
  Just more spam and scams than they can handle. 0.5% of all the shit on the Internet is real content. And that stat is from 2015. Their own fault for monetizing every imaginable search query. It has provided the incentive to ever spammer and scammer on the planet to mass produce copious amount fake interlinking pages massively larger than actual Internet.
  - jodrellblank 3y ago
    Vernor Vinge was wrong about humanity's vile offspring being stock trading bots, and the sci-fi authors were wrong about Grey Goo being a nanobot scenario. We're awash in grey goo vile offspring right now, consuming resources at an increasing rate and turning everything bad - and it's information; SEO spam and content farms and LLM waffle. Pretty soon the only way to live will be an Encyclopaedia, text books, and cookbooks from before 2010.
    - gsatic 3y ago
      Its not a static world. Things are always changing. I have some faith Google(the ad tech part) will start crumbling. Chat GPT has given them a jolt. And there are more jolts to come. Not less. From the Regulators. From the advertisers who know most of the views they get are from bots or totally worthless. And that they don't need to be spending what they are spending.
      - pixl97 3y ago
        I'm not sure this fixes anything. GPT/LLM will be able to produce monumental amounts of reasonable sounding bullshit. Google may crumble and fail, but all we'll be left with is small islands of sanctuary in an ocean of bullshit.
pawelduda 3y ago
Maybe they ran out of disk space
- deleted 3y ago
  [deleted]
- foobarbecue 3y ago
  Sorry guys... I've been uploading all of every video I take to youtube... I should have known this would happen.
  - ElFitz 3y ago
    That can’t beat those using YouTube as a low-cost backup platform. For those who missed it: https://gizmodo.com/backup-data-on-youtube-hack-white-noise-aka-isg-1850261527 https://gizmodo.com/backup-data-on-youtube-hack-white-noise-...
    - marcod 3y ago
      It's not very efficient?!? I am shocked!
philipov 3y ago
The site appears to have been hugged to death.
butler14 3y ago
Difficult to comment without URLs, there's too many potential causes Unfortunately "[being] a web designer since 2016" is generally not helpful for SEO But certainly setting up Google Search Console, submitting a sitemap, and 'fetching' your homepage on day 1 is good practice, especially for a brand new website
- dskrepps 3y ago
  If you're mass-producing low-quality sites you would do that on day 1 with each of them. So it likely doesn't weigh positively as much as we would hope.
- user_666 3y ago
  That is my blog post. I did that with the convention site, yes. It still wasn't getting indexed.
  - BryanBigs 3y ago
    Did you build any links to the site? Links=Votes=Trust in Google's eyes....for better or worse. Even a tweet or Pinterest pin?
    - user_666 3y ago
      yep, all of that!
- CTmystery 3y ago
  I can also vouch that what the author is saying is true. I've started publishing my notes, and I found that after three weeks none of them were indexed (and yes I submit a sitemap through google webmaster tools). I have since started using the URL inspector tool and adding them one by one. That does work.
paxys 3y ago
[flagged]
- gwbas1c 3y ago
  Sometimes people like to experiment on their personal / side projects.
  - hgsgm 3y ago
    Me too. But I don't claim to be an authority on web internals.
- dom96 3y ago
  The traffic isn't even that high. It boggles my mind any time a website cannot handle it.
  - Ekaros 3y ago
    I always wondered. How did we manage to run anything lets say 20 years ago. We should have multiple times more memory, cpu, bandwidth available... Or are we just that many magnitudes more inefficient?
    - timw4mail 3y ago
      Yes, it is magnitudes more inefficient, with large JS payloads, slow APIs, and underpowered cloud hosts.
    - eterm 3y ago
      Websites often went down under load 20 years ago too but people were far more quick to think it was their own connection on the blink. Yes, sites were much better engineered too, with smaller and less rich and less dynamic payloads, but don't count out a change in expectations along with it.
  - andersa 3y ago
    How much traffic could that be? Given the number of comments on this page, I can't imagine it being more than a few thousand clicks? So why would any site go down?
zolland 3y ago
[flagged]
- KomoD 3y ago
  That's just on archive.is
tyingq 3y ago
It seems obvious to me the importance of organic results has fallen over the years. There's plenty of queries where you have to scroll down a fair amount to even see them. Pushed down in favor of ads, various widgets, content sourced from Wikipedia and published on google's urls, etc. Things that sit below the fold get less overall money, time, and attention.
- hackernewds 3y ago
  With generative AI search results, soon you won't even be able to know whether your site was used for the result or not. Lots of no click queries, resulting in no traffic for the publisher
  - userbinator 3y ago
    The much more worrying fact about AI is you won't even be able to know whether the information you're getting is true. I always scroll past the crap at the top to get to the actual site results.
    - danielheath 3y ago
      I haven’t been able to estimate that with confidence in a decade; SEO blogspam has seen to that.
      - nicbou 3y ago
        The content is true to the extent that a copywriter paraphrased another piece of content that is more authoritative. My work is getting paraphrased a lot, and they usually get the gist right, although they have zero clue of what they're talking about.
    - Karunamon 3y ago
      Given the web as it is today is infested with clickbait, "native content", clout-chasing, undisclosed sponsorship, and other such pathologies, I'm not convinced that the fear of AI making truth more rare is rational.
skilled 3y ago
Things are pretty bad with Google in this department. Over the last two years they have been constantly updating and changing their algorithm, there are now updates such as Core, Product Reviews, Helpful, Spam, Link Spam and God knows what else. And they are rotating them 24/7. Literally. You could be up 20% one day and down 40% the next and you are none the wiser as to why. If you are a small time publisher with no budget or no craftiness to attract links from other sites - you are pretty much doomed. I have a few sites that are in the 200k monthly visits range (from Google) and Google only fetches the homepage/feed every four hours or so, sometimes it takes longer than that. It’s a lot different from what it used to be.
jmclnx 3y ago
My site seemed to have come up fine. I wonder if it may be related to to the wayback machene "https://archive.org/web/ https://archive.org/web/" ? My site is on it has has been there for a while. I also have these on my WEB pages: <head> <META NAME="DESCRIPTION" CONTENT="MYTITLE"> <META NAME="KEYWORDS" CONTENT="COMMA SEPARATED LIST OF SIBJECTS"> <META NAME="AUTHOR" CONTENT="MYNAME"> <META NAME="RATING" CONTENT="General"> <META NAME="ROBOTS" CONTENT="index, follow"> <META NAME="REVISIT-AFTER" CONTENT="90 days"> </head> Maybe that will help out ? But I do notice google searches are not as good as years ago, and duckduckgo is close or maybe better than what google was.
metalrain 3y ago
Should Google index every website? I think answer is historically been yes, but more and more it's becoming no. Users of Google search want the best results, not all results. Customers of Google want their ad or site visible, not all sites.
- Ekaros 3y ago
  Funny thing is that very often I am not seeing the best result. Or even good result, but some stolen or SEO optimised junk... So to me it seems they are missing good content, not the bad.
  - hospitalJail 3y ago
    I have a website that is #1 when it comes to the problem I'm solving. No one comes close. Its mostly due to, its a money saving thing for consumers, and that isnt really profitable. Its low hanging fruit, and I have the best website for it. Nothing really comes close, most alternative websites make mistakes in their advice because they are using feelings rather than millions of entries of data. Anyway, if you search specifically for my most popular metric, you will always get my website. If you google 'cheap X', you will get inferior websites. Even with SEO optimized, there are just bigger websites that are friendlier with google, linked by other websites, or it could be better SEO. Whatever the case, it makes me wonder what kind of websites I miss because I use google.
    - peter422 3y ago
      If the site is so good other people should naturally link to it over time and the situation will work itself out.
      - hospitalJail 3y ago
        It does, but the giants are just so much better at the simple phrase 'Save money on X'
- deleted 3y ago
  [deleted]
margarina72 3y ago
well, I can't say for everyone but I did notice that google stopped indexing a few sites I manage. No technical reason, no error, just says "Not indexed - reason: Discovered - currently not indexed" and that's most of the content of the site (nothing fancy mostly technical articles, and others notes) Unclear why this is happening.
- gscott 3y ago
  They won't index it until you have enough links coming into it.
flenserboy 3y ago
Google is an advertising & data-collection business, not a search company. Search is their on-ramp for attracting sources of data. If your website is not worth much on either side of the business, you'll be a low-priority target.
- frouge 3y ago
  Yes, Google is an ad-engine
eth0up 3y ago
I have a year old WP site that showed up on Google, but wasn't indexing all pages. I submitted the .xml sitemap via https://www.google.com/ping?sitemap="url" But that didn't seem fully effective. I then succumbed to uploading a verification file to the root directory (requires Google account) and resubmitted an inddx request. Within 12 hours a Google search of Site:"my url" yielded all pages. I'm terribly rusty with websites, hence my use of WordPress and willingness to taint my root directory with Google files. I do notice that exact, relevant queries in quotes still show no results for some content. Much to relearn.
- franze 3y ago
  was the sitemap.xml located in the root of the domain? https://www.google.com/ping?sitemap= https://www.google.com/ping?sitemap= only works if there is a previous root reference (i.e. via robots.txt or via GSC submit) of the sitemap.xml right issue, as from the original spec a sitemap can only submit URLs of the same or lower URL hierachy to Google. or there is a root reference to the sitemap.xml
  - eth0up 3y ago
    I don't remember the initial loc. It's now in the root dir as mentioned. However, I'm still not pleased with the results, but perhaps it takes time. I'm sure backlink or two wouldn't hurt.
userbinator 3y ago
On the other side, I've also noticed it appears to be aggressively pruning its index in the past few years, so the fact that it's crawled your site doesn't mean it's necessarily searchable either. Another "bug" that seems to manifest quite often: if I search for a specific phrase or unique word on a page that I found in a SERP, so I know it's crawled that page, it often doesn't return that page either. Add to that the automatic CAPTCHA-hellban you get if you use "site:" in anything more than a tiny amount (and the one you still get if you search "too much"), and I realise that there's increasingly huge amounts of information out there on sites that Google may have crawled before and knows about, but doesn't want to show me for some reason. I remember it used to be much easier to find information about obscure topics even if it meant wading through dozens of pages of SEO spam; now it's nearly impossible for anything but the most vapid of queries.
- nohuck13 3y ago
  "Add to that the automatic CAPTCHA-hellban you get if you use "site:" in anything more than a tiny amount" Source? This would be worrisome.
  - josephcsible 3y ago
    FWIW, I've personally experienced exactly that happening too.
  - pteraspidomorph 3y ago
    I personally have been surprised to find myself CAPTCHA'd out of google search recently. No idea what's up with that. Regular commercial ISP, no VPNs.
  - tyingq 3y ago
    Anecdotal, but this happens to me a lot, and not just with the "site:" operator. Generally using any of the advanced operators seems to set it off. Things like inurl:, intitle:, etc, trigger it also. Not every time, but after a few times. From a normal ISP connection, no VPN, even while logged into Google, etc.
  - 3y ago
api 3y ago
I must once again plug Kagi.com. They're not perfect but are an actual search engine and I now use them as my primary. I am not affiliated with them.
- arbitrage 3y ago
  It appears you have to create an account before you see any search results. No thanks.
  - allarm 3y ago
    Umm, subscription is actually the way Kagi monetises, instead of selling ads. This is exactly why you should consider it.
    - mrweasel 3y ago
      The complaint is still valid. I have subscriptions to a few sites, mostly news. I get why I need to login, otherwise how would they know that I've paid. It's still a pretty poor experience, I pay for a service, and now I need to fiddle with the settings in my browser so the site I pay for can remember my credentials. I get that part of the issue is that I keep a weird privacy focused configuration in my browsers and delete cookies when I close the browser, still it results in poor user experience, for a service that I pay for. I don't have any good suggestion on how to fix it, but it's is valid to complain about having to sign in.
      - efdee 3y ago
        It's a poor user experience because you break it on purpose. Whatever other way they can come up with to permanently identify you, you will block that in the name of privacy and keep complaining.
        mrweasel 3y ago
        I don't really care about privacy from a company that already have my credit card information, and real name and address in most cases. I care about privacy from "the others", and the paid services is collateral damage. It does appear from another comment that Kagi actually thought about it and allows a token in the search URI, that's pretty neat.
hackdojo 3y ago
The site seems to be down (likely due to the site being on HN top page) As for actual indexing, we found the Indexing API to be the most reliable. https://console.cloud.google.com/apis/library/indexing.googleapis.com?project=pse-development https://console.cloud.google.com/apis/library/indexing.googl... The caveat is that you can only request Google up to 200 URLs to index. It's also technically not the most 'correct' way to have your pages indexed. (supposedly used for job page updates, etc)
voytec 3y ago
Hugged. Version from May 31: https://web.archive.org/web/20230531062024/https://natehoffelder.com/blog/google-no-longer-automatically-indexes-websites-wtf/ https://web.archive.org/web/20230531062024/https://natehoffe...
- deleted 3y ago
  [deleted]
- moontear 3y ago
  Thank you
  - deleted 3y ago
    [deleted]
- user_666 3y ago
  Yeah, sorry about that - I didn't realize my story had been posted ehre until just a few minutes ago.
- alpb 3y ago
  How's it even hugged while being behind Cloudflare?
  - luuurker 3y ago
    "cf-cache-status: DYNAMIC" The page isn't being cached. Cloudflare doesn't cache pages by default as that would create issues (logins, comments, admin sections, etc). They cache static files (images, css, js, etc), but that's it. You need to set it up yourself and have ways to bypass and purge the cache.
    - account42 3y ago
      > Cloudflare doesn't cache pages by default as that would create issues (logins, comments, admin sections, etc). They cache static files (images, css, js, etc), but that's it. You need to set it up yourself and have ways to bypass and purge the cache. Or they could just rely on Cache-Crontrol headers (and friends) instead of requiring CF-specific configuration.
kubanczyk 3y ago
Hugged to death. Appreciate the irony of reading it from Google's own cache. https://webcache.googleusercontent.com/search?q=cache:https%3A%2F%2Fnatehoffelder.com%2Fblog%2Fgoogle-no-longer-automatically-indexes-websites-wtf%2F https://webcache.googleusercontent.com/search?q=cache:https%...
digitcatphd 3y ago
I don't think this is a bug. After all, why waste resources indexing pages 2 .... n when nobody even goes there? If something accrues signals outside of Google naturally, then I am sure it will get indexed. On this note, I don't want to be another 'SEO is Dead' person, because it is never dead, it just changes, but I do suspect over the longer term, links will become less relevant and Google will take a more human-curated approach to ranking selection using LLMs.
Avamander 3y ago
They've been flooded with large amounts of new domains/sites hosting generated garbage content. I suspect this is just a barrier to slow it down until they figure out how to detect such spam better.
franze 3y ago
well, can't really comment without URL but one thing I found majorly annoying is who sticky "noindex" has become. used to be that you could put your staging on noindex, put it out there, do a lot of external testing tools (pages speed insights, mobile friendly tools, webpagetest.org, 10000 other tests that can easily be done on the open internet) and send the page around, and then on publishing day just remove the noindex, trigger recrawling and be indexed and visible in Google in no time. now it takes weeks to month for the noindex to vanish. even after google has crawled the pages again and again as visible in the logfiles, the pages stay noindex even though the information is long gone. google, the lazy monopoly strikes again.
- user_666 3y ago
  I've noticed this as well. I now use a different slug or domain for the development page(s), and change it when the page goes live.
gniv 3y ago
Indeed this very article doesn't come up in searches like [ddg site:natehoffelder.com] or [noindex site:natehoffelder.com]. And the article has been up since at least May 30. So yeah, looks like an outage at Google on the crawling side.
- tensor 3y ago
  When I go to the article I get some sort of cloudlflare "check" intermediate page. Could that be interfering with indexing?
  - user_666 3y ago
    No, that was because I thought my site was under attack (this was before I learned the story was posted here).
bradley13 3y ago
Can confirm. While I don't do many websites anymore, I used to, and have since Google was the new kid on the block. I recently put up a new site, and...nothing. As the author said, I finally had to force each individual page to index. Harder to quantify, but it seems that search results are also worse than they used to be. It's possible that Google is doing some internal cost cutting...
danhak 3y ago
I'm experiencing this right now with my site https://www.dropspotter.com https://www.dropspotter.com I've done everything right according to Google, set up the search console, uploaded a site map, addressed all mobile usability issues. And yet only a tiny fraction of my content is being indexed. I'm a bit at the end of my rope here as I've poured a year into this project and getting a historically normal amount of search traffic may be the difference between this project being viable or not. The most frustrating part here is having zero visibility into what's going on.
- soared 3y ago
  Your site has very little “content” and is just product listings which looks like a million other garbage sites google sees. Great for ux to have the product listings users want, but the crawlers need some plain text to read to understand what you’re site is about. I’d add a paragraph, hero image image, cta/etc at the very top explaining what your site is. Additionally you need a menu at the top and footer at the bottom with links to additional content-only (not products) pages - IE an about us, where do we source data, etc. Even 2 blog posts would help a ton. Do not stuff them with keywords but be sure to use the words that are common in your niche of the industry so your site gets associated with industry sites. From googles eyes the difference between a scammy online ecom site and your site is hard to see! (Even if your site provides legitimate value to users). You can try posting a link to it on various digital Marketing subreddits (not the like “rate my website” ones) to see if you can get more feedback - I haven’t done that in years though. Edit - also didn’t realize clicking product links takes you to external sites. That’s a tough site for google to ever understand correctly since you have so little content and the best possible outcome of a user visiting your site is that they leave it. Maybe set it up to have each product link to your own page for it, maybe with price history from camelcamelcamel, links to the product in tiger sites, generic info about what site it’s listed on, or just crawl the description at the vendors site.
  - danhak 3y ago
    Thank you for this extremely helpful response
- 3y ago
awesometech 3y ago
I think I'm going to try using DuckDuckGo for a while and compare, might even try Yahoo search.
- nerdchum 3y ago
  I found Brave search to be awesome. search.brave.com It's working on this idea called Brave Goggles where you can click on a goggle and it tweaks the algorithm to prioritize that type of content or specific sites get higher ranking It's pretty cool. All their results have been fantastic as well for me and it's supposed to be private as well. It's pretty much replaced google for me without even noticing.
  - roter 3y ago
    You could do both: use DDG but preface search with !brave as needed.
- user_666 3y ago
  I mainly use DDG, and this is part of the reason why.
  - efreak 3y ago
    The problem is that Google actually has support for quotes, even if it's not reliable. Try searching for short song lyrics or poetry, or pretty much any obscure quote on ddg and you can't find it. Putting individual words and phrases in quotes does nothing on ddg unless they suggest it for you...in which case it often still doesn't work
aa_is_op 3y ago
It will not index your website unless you register a Gmail address and then a Google Console account.
- lispy4 3y ago
  Also, Gmail is not playing nice with non-Gmail email accounts. If it's not coming from a large corporation, Gmail is filtering it.
nomadness 3y ago
kicked out of Google's index for no apparent reason, only to be thrown back in just as randomly. It's like a never-ending rollercoaster ride! The GSC comes to the rescue by helping to get important pages indexed fast, but guess what? You can only submit a measly 10 URLs per day! Ugh, talk about frustration!.. :-/
_joel 3y ago
It's a shame the website looks so broken (in ff)
TheCaptain4815 3y ago
Once again the black hats will come out on top. As someone in that industry, there's always a new trick or method to getting your sites indexed en mass before others. If you don't have time to find it, someone in a 2nd world country (probably India) will offer the service for relatively cheap at BlackHatWorld. On a side note on the SERP (search engine results page) of the foreseeable future, I don't accept the doom and gloom other SEOs have. Google has shown it will give credit to the top 3 links, and to be honest pretty visible credit. This will shift the target from "front page" to "top 3" sort of like Local Seo Map Packs. Will make difficult keywords more difficult to get clicks for, but long tail keywords could be easier to rank now as a #3 ranking could garner as many clicks as a #1 ranking.
danwee 3y ago
Google Search is in decline for the users who know how the internet works. For my mother, the internet is still Google (and would remain like that no matter what). For me, for some friends of mine, and for many of my colleagues who know more or less how things work on the web, Google Search is just in free fall: we use it as last resort, but as other search engines (or other tools, like ChatGPT) improve over time, Google Search would just disappear from our bookmarks. I know that many of you would say that it doesn't really matter what "hackers" think about Google Search, that all that matters is that the majority of the non-tech-savvy users still think Internet == Google. Well, let's talk again in 5 years.
- okdood64 3y ago
  I'm pretty sure you could find doom & gloom talk about Google on HN 5 years ago.
  - pil0u 3y ago
    Only 7 years ago, the world was astonished with the release of AlphaGo. This was an algorithm dedicated to one very specific task of winning a game. The most recent game being played is language, a game that Google has played alone, until now. Doom & gloom talk about Google 5 years ago was probably unjustified and rightfully dismissed. Today, I'm not so sure about that.
  - zarzavat 3y ago
    That’s because it’s been progressively getting worse for years. For example, when they launched Google+ they stopped supporting the + operator in favour of “quotes” and people complained about that. Of course now the “quotes” don’t work either. Google used to be amazing. If you remembered a set of words that were on a page, you could enter those words and it would find matching pages. It’s been broken for years and keeps getting more broken.
    - okdood64 3y ago
      Broken? https://www.google.com/search?q=%22may+have+been+less+attractive+to+candidates+heavily+focused+on+the+job+title%22&oq=%22may+have+been+less+attractive+to+candidates+heavily+focused+on+the+job+title https://www.google.com/search?q=%22may+have+been+less+attrac... I just took a random snippet from Gergely's blog and using quotes I found it. Look there's many issues with Google Search in 2023, but these nitpicks that aren't neccesarily true aren't productive to the conversation.
ahstilde 3y ago
If you're not in the top 5 results, there's no difference between not being indexed vs being indexed. If you want to be indexed, submit your pages to Google explicitly through a GA tag, a Google Search Console account, and a sitemap. If you're publishing many, many pages, you might see a benefit with the indexing API: https://developers.google.com/search/apis/indexing-api/v3/quickstart https://developers.google.com/search/apis/indexing-api/v3/qu... > I have been a web designer since 2016, and before that I was a blogger for 6 plus years. I have been deeply interested in Google SEO since 2010, and in all that time I’ve never heard of Google not indexing a site. Google crawls sites it assumes will have useful content. If it doesn't have that assumption for your site, you need to feed it.
- ris58h 3y ago
  > If you want to be indexed, submit your pages to Google explicitly through a GA tag, a Google Search Console account, and a sitemap. What if I don't want to put any Google-related stuff on my site? I have a GitHub pages site with a very unique topic and it's still not indexed by Google (the site is available at least for 8 month now). I've checked Bing and my site is on the 4-th place on the results page.
  - dawnerd 3y ago
    Use a sitemap, that’s not google specific at all.
  - ahstilde 3y ago
    > What if I don't want to put any Google-related stuff on my site? Then you're choosing to make it harder to be platformed by Google. Regardless, a GSC account and submitting a sitemap do not require you to serve any Google-related "stuff".
- user_666 3y ago
  That's my blog post. The convention site in question had a GA tag, and it was in search console, and I had submitted a sitemap. Search Console knee the site had 9 pages in the sitemap, and had only indexed the homepage. The other pages weren't rejected, declined or 404ed - they were simply ignored.
peter422 3y ago
There are an infinite number of URLs on the internet, Google obviously cannot crawl and index them all. They choose what to crawl and index based on mainly site reputation and inbound links, just like they used to 20 years ago. If they aren't crawling or indexing your site, then link to it from a high reputation place.
PaulHoule 3y ago
Google sees “putting your site online, getting it indexed, having it appear in search results, and receive traffic” to be competitive with “pay for advertising and premium placement.” The long term answer will be some kind of “Google Blue” where you have to pay for any placement, not just premium placement.
nforgerit 3y ago
So are we all now officially complaining about that it stinks where we've SEO-sh*tted for two decades? Good, then let's get rid of this ad network dystopia which never worked as advertised (they show me special offers for a frying pan, 2 weeks after I ordered one. Laughable ya folks). Replace it with great authentic inspirational content and people will throw their 2 bucks at it.
- teddyh 3y ago
  Too bad that the powers that be has decided that payments must be identified, so such a new internet will by necessity not be anonymous, but instead tracked and identified in every aspect.
  - nforgerit 3y ago
    There's a difference between me deciding which payment method I use for service X and having service X decide into which tracking networks they put me in. Additionally, Mullvad showed how to use cash via regular post to pay 100% anonymously for service X.
    - teddyh 3y ago
      Oh, there are ways to do it nicely! But they won’t be used, and won’t be available.
- executesorder66 3y ago
  What's the point of censoring only one character from the word "shitted"? Both you and everyone that reads your comment know exactly what you are saying, so it's not like you are shielding anyone from anything. And neither is there anyone who is going to punish you for saying a naughty word on the internet. What are you afraid is going to happen if you write "shitted"?
  - nforgerit 3y ago
    My mom's watching and soap is just not my kind of taste
ovulator 3y ago
I've noticed this for a side project I run. https://cfbpedia.com/ https://cfbpedia.com/ I upgraded it's server a while back and unknowingly borked it's SSL redirect. I corrected it and Google hasn't picked it up again since despite my fiddling in every way in their Console, still just "not indexed". If you search for it directly in Google you will find links to it, but never the site itself. I figured I was doing something wrong and didn't care enough to fix it, but it definitely seems like Google is making some sort of attempt at pruning.
ravenstine 3y ago
Maybe the concept of "web search" needs to be reimagined all together? Perhaps the idea of creating a master index of the content of multiple pages on nearly every site is the thing that is not sustainable. Instead, "search this site" needs to be made great again. Individual websites could manage their own search in a way that complies with a standard API that can be consumed by meta-search engines. Rather than indexing pages in the traditional way, meta-search engines instead use a heuristic or AI model to decide what sites are going to have the kind of information you are searching for, perform your query on theirs, and return the aggregated results to the user. The less the algorithm understands the significance or meaning of the query, the more generalized its approach can be. For instance, if it thinks that you're searching specifically for opinion-based content that will appear on blogs and forums, then it will target a federated search engine that indexes those things specifically. But if you are searching for information on making beer at home, it will know to target and weight the search engine on brewersfriend.com and homebrew.stackexchange.com. Although this sounds not that different to how search currently works today, remember that this idea is about having search become more federated and more standardized, and for meta-search to select federated indexes rather than own a god-index. A user of a meta-search can pick and choose what indexes they want available in their searches in case they find any of them to be either superior or particularly problematic, and the meta-search can optionally adjust its understanding of a particular user's queries. The way I see it, traditional search will continue to decline in part because it's not sustainable, but also in response to AI allowing them to become "answer engines". Although a lot of people do want an answer engine, this isn't for everyone. I think there will always be a market for people looking for content on specific webpages. Whatever that thing is that someday snatches that market away from The Google, if it's going to be successful, won't survive on the current concept of what "search" is.
codingdave 3y ago
I never thought they did it 'automatically'. My impression was that they always had to have found it via a link from another site, or that the site owner had to submit the site to Google. It isn't like Google is seeking out all registered domains and checking each one to see if a site suddenly appeared on it. So if someone registers a new domain, puts a site on it, doesn't link to it from anywhere else, doesn't submit to to Google... and yet expects the site to be found by Google, that is just not a reasonable expectation.
- rightbyte 3y ago
  I think they abuse Chrome or their 8.8.8.8 dns lookup somehow too. Sites certainly show up without links.
SL61 3y ago
I've seen this on a site that I built for a freelance client a couple years ago. At first they only indexed the homepage and some sporadic articles. Eventually one specific article was linked on an .edu website, and now that article is reliably indexed as well. If I manually request that they index a page, it always succeeds and shows up on Google within a few days, but the page gets pruned from the index within a month or two. The weird thing is that within the past couple years I've also seen unwanted indexing, e.g. low-spec staging servers getting wrecked by Google crawler traffic. I don't think the "no automatic indexing" treatment is standard for every site, so there must be something that triggers it, but I've spent a long time unsuccessfully trying to pin down the cause.
underlipton 3y ago
Have they fixed "pieces of me is actually a good song reddit" yet? (Google it.)
coldtea 3y ago
Perhaps they're not even trying anymore?
ltbarcly3 3y ago
Couldn't read, cloudflare 'verified my safety' and then hung for a couple minutes and I'm giving up.
- sshine 3y ago
  I'm in the same situation.
dbg31415 3y ago
Google has to know about the site before it can index it. Set up the sitemap, then link the sitemap in from Google's Search Console Tools, and install Google Analytics. This will help Google pick up that your site exists. Make sure your robots.txt file is configured to allow crawlers. Make sure your pages aren't inadvertently NOINDEX'd. SEO isn't as relevant as it used to be, but all this stuff should be part of your QA and pre-launch checklist. Get 90+ Lighthouse scores. https://developer.chrome.com/docs/lighthouse/overview/ https://developer.chrome.com/docs/lighthouse/overview/ Run your pages through WAVE or some other accessibility checker. https://wave.webaim.org/ https://wave.webaim.org/ Run your home page through Security Headers or some other front-end security audit. https://securityheaders.com/ https://securityheaders.com/ Set up uptime monitors on every page that gets more than 1% of your traffic... check page load speeds and HTTP response codes -- you never know when a WAF or some other system will get mucked up.
the_duke 3y ago
My personal guess: with ChatGPT and other locally runnable LLMs being commonplace now, Google just can't keep up anymore. There must be millions upon millions of SEO spam pages being created and updated every day. They lost the battle, at least for now. This also lines up with Google Search results becoming increasingly worse over the last few years, but the last months in particular.
- zarzavat 3y ago
  And who invented LLMs, transformers and modern AI? Google. There would be no ChatGPT without Google’s decade long campaign to create it.
- Exuma 3y ago
  Hmm, this is the best guess here IMO. This person did say it started occuring within the last year though. I just created a site a month ago with 2000+ pages and I did notice only 5 were indexed, which was quite annoying. I just suspected it was because I didn't have enough backlinks yet so it didn't want to waste time on it... interesting it may be what this original article is about. But yes, I 100% bet its because all new sites now have an enormous barrier with the sheer amount of fake content that can be generated.
boosteri 3y ago
Why does Nate Hoffelder need to review the security of my connection all of a sudden? It didnt seem to bother them an hour ago /s
- user_666 3y ago
  That is Cloudflare. So many people were clicking on the link from here that it inadvertently DDOS-ed my server. I had to turn on the "under attack" feature in Cloudflare just to get the other sites running again.
mo_42 3y ago
Google is part of a profit-maximizing company. Slowly, they are transitioning from showing relevant results and some ad in a separate box to results that maximize profit directly because they’re sponsored or whatever. The founders even explained it in one of their papers. Speculation: Crawling the web is expensive. So it makes sense that they decide for each crawled page if it's profitable to put that into the index. In the long run they might not crawl the web at all. People will just pay them to get into the index.
fps_doug 3y ago
As a random data point, I registered a new domain about a month ago and didn't do more than add a simple landing page yet, and it is properly indexed by Google.
mattbee 3y ago
Yup this confused me the other day - tried to search for an article I'd written for Dr. Dobbs Journal in 2001 on a very specific topic. No amount of keywords including my name, technical keywords etc. could find it. I assumed that meant it had been deleted from the web; the publication that had disappeared 10 years ago. Then saw I'd linked to it from my own site, and the link worked just fine :facepalm: It's weird to see Google forget the kind of content I used to go to them for in the first place.
giancarlostoro 3y ago
Google has been so weirdly different and off lately. I miss when I could type anything and get exactly what I was looking for. Those days died at least ten years ago. Sidenote: I wish Google would lower the ranking on websites who push for a mobile app and degrade your web browsing experience. There's no point in even getting them as a search result. If I wanted something from an app, I would have just gone on the app store.
ChatGTP 3y ago
Risk or not, I'm not sure why Sam Altman's views are really so important in this discussion. He is a CEO. Period.
WirelessGigabit 3y ago
The quality of searches have gone down so much. A lot of times these days you end up in these pages which have a page for every iteration (not the right word) of a sentence, pointed to their page full of 'answers'. Who is to blame? Google. For years they have pushed people to have slabs of text. Now I need to read through somebodies life story and how they stubbed their toe against a rock while hiking in the Sonoran foothills followed by a recipe for pulled pork tacos.
flumpcakes 3y ago
I think search results have become worse over time for three main reasons, firstly the amount of absolute drivel that's published on the web, made doubly worse with AI written drivel. It's honestly shocking. Secondly, the web has become more centralised and people use aggregators more than individual sites or services (facebook, reddit, etc.). Thirdly, the internet has become 'safer' in a lot of ways and seemingly the search engines scrub what is probably the majority of results from ever being returned. It used to be quite easy to find a pdf of a book on some open web server from a google search alone, now it's nearly impossible. And I don't think that's because people's security hygiene has improved.
koromak 3y ago
I don't understand how Google fails so miserably at doing its main job. I can't wait until I never have to use it again.
- timbit42 3y ago
  It's main job isn't search -- it's main job is advertising.
masswerk 3y ago
Another thing, I noticed: a weird prioritization of certain sources. E.g., I've a website that is now up for 24 years (there's even some content older than this, which has moved there), it has my name on it and my name is in the meta data, and – quite naturally – it used to be the top result, when you googled me. Recently, I discovered that the top results are now some university pages that are not maintained, or even have never been set up. Which really undermines the purpose of the site. (Notably, the site has quite a high number of reported Google search hits, has rather perfect Lighthouse scores, and SEO ratings are high enough that I get SEO related requests – which are happily ignored – on a rather regular basis. It scores about rank 60K in the "Majestic Million". And the site enjoys a few updates/fresh content a few times a year. So this is not a case of obscurity.)
lispy4 3y ago
Google has also begun aggressively filtering domains from sending email to Gmail users. This rolled out last November but they've ramped it up this past week to the point that many "indie" domains are 100% blocked from emailing Gmail accounts.
- nubinetwork 3y ago
  I haven't had issues sending emails to Gmail from my personal domains... but spamhaus/sorbs blocks Gmail on the inverse because people use it to send so much spam.
iza 3y ago
I've noticed my newer GitHub repos are not indexed by Google for weeks, while on Bing and DDG they show up very quickly.
renegat0x0 3y ago
That is why I wrote [1] for myself. It stores links in database, which I can query. Everything is later on exported, like in [2] and [3]. I can browse history, I can find useful data. I do not say it has replaced google for me. It is a nice addition that helped me gather data I encounter on the Internet. It is a link database, at first glance resembles Reddit clone, but my focus is on creating link database, not on providing social media experience cancer. Links: [1] https://github.com/rumca-js/Django-link-archive https://github.com/rumca-js/Django-link-archive [2] https://github.com/rumca-js/RSS-Link-Database https://github.com/rumca-js/RSS-Link-Database [3] https://github.com/rumca-js/RSS-Link-Database-2023 https://github.com/rumca-js/RSS-Link-Database-2023
jojohohanon 3y ago
Checking if site connection is secure And I left. Irony all exhausted
ChuckMcM 3y ago
My guess is that Google doesn't care (not that they ever did but now less than they did before). The simple math is "Google makes no money by indexing your website." The easiest way in the past to get reliably indexed by Google was to put Google ads on your web site. Then they have skin in the game so to speak. "Long tail" websites, those that are very niche, not especially linked outside of a few folks, have only ever been there to impress people using Google into believing they index everything (which they don't, and frankly haven't for at least the last 10 years). Over the years, as all of the ways that people who hosted a "search engine", exploited the "digital exhaust" of the people who used it, and sold that information for profit, have been revealed, well regulations and laws have been enacted to reduce or eliminate the egregious (and usually more profitable) ones. That and advertisers tired of paying for "engagement" and seeing few sales or conversions, the economics of running a search engine as this high margin, make money hand over fist, days are slowly winding down. That Bing has (for now at least), continued to index these sites, will just accelerate Google's decline. When I was helping to run Blekko, people would look at our curated results and be super impressed at how much better they were than Bing or Google results, but then they would search for their cousin's minecraft blog and wouldn't find it and lament that they couldn't possibly use it as their "daily driver" because if it didn't have their cousin's website in it, how could they know what they weren't seeing? Blekko tried really hard to make the argument that if you made a list of the minecraft blogs you followed as a Blekko user we'd index all of them, and if everyone did that for their favorite stuff then the index would fill up with good web sites and not be riddled with junk. But sadly we couldn't get them to internalize that and they weren't willing to create an account to have a better web experience. Perhaps we were just too early but still it is a weird thing.
- marginalia_nu 3y ago
  I think it's important to realize that search engines both bring websites to people and people to websites. The genius part of Google's position in the market is that they both sell ads, and then direct people to the places where the ads are sold. This is also a major contributor to their search engine spam problem; they can't well penalize ads without undermining their core business model. Although I don't think any search engine has ever indexed or will ever index everything. That doesn't really make sense. Realistically I think maybe 1% of the documents online are ever going to be a good search result for any query ever. Internet search is all about being judicious about what you index.
ffpip 3y ago
For some reason it refuses to index my site(s) until I link the site to my Google account in search console. I had organically placed a lot of links (on my GH, Linkedin profiles, no spam anywhere). It did not index even the links for over a month, despite me searching the exact domain multiple times. I guess doing it via search console just showed it the exact domains it has to crawl, making it's job easier
Melatonic 3y ago
Another reason I am glad I paid for Kagi hahahaha
zmix 3y ago
Each time I search for something, the first two, three pages, maybe even more, are just shops and affiliate link sites, that offer the thing I want information about. At least, if thing is a product. Not a single forum reference. I need to add the keyword "forum" in order to get such results. To me that is typical corporate degeneration and decline. Too big to fail, to trashy to be relevant any longer, instead, however, omnipresent.
Animats 3y ago
So this means you now must have a contractual relationship with Google to appear in Google search results? That raises major antitrust issues.
EMCymatics 3y ago
This should be a bigger deal
hannob 3y ago
I have recently started a new web project ( https://industrydecarbonization.com/ https://industrydecarbonization.com/ ) where I am relatively closely following how it does on search engines, and I can't say that I share those experiences. It took a few weeks until google noticed that it's a page with relevant content, but that's kinda expected. But once it did I feel Google is indexing my pages extremely fast, so fast that I have been wondering how they're actually doing this. I post new content on various social media sites, and my best guess is that google gets some of them as direct feeds that they check for interesting links. Google does not support indexnow, and as far as I know also no similar feature (except manually via the search console), so I'm not in any way directly submitting my content to Google.
deathanatos 3y ago
> and in all that time I’ve never heard of Google not indexing a site. […] Google used to just index every site whether you wanted them to or not Google can only index what they know of directly, or can indirectly discover e.g., by links. I have unindexed sites because, well, they're not linked to on the public Internet. I can pretty easily see a 2 month old website being outside that; the question is does the rest of the Internet know about it? (And, the public Internet. A shared link in a Discord channel also isn't visible to search engines…) Some of the examples contain statements that indicate Google could or should know of them, but not all of them.
paulrpotts 3y ago
I have a personal web site with almost 1500 pages containing blog posts, newsletters, reviews, and articles going back to the late eighties. I've been slowly consolidating all my writing on this site. For years I had kind of assume that if I build it, (eventually) it will be discoverable via organic search. I recently determined that I had to put a Google ID on it and a sitemap file. So I've found scripts, imperfect thought they are, to crawl everything and make a sitemap. Google has indexed some of it, but seems to be stalled out, with the bulk of the pages stuck in the "Discovered, currently not indexed" state, and now I'm not sure if some of the best content will _ever_ be searchable via Google. Really disappointing.
thedailymail 3y ago
It should be concerning for google that, 25 years since PageRank, search is once again ripe for disruption.
yoru-sulfur 3y ago
My website seems to have no indexing problems. I would be shocked if I had literally any other sites linking to mine (Small personal website), so at least from my perspective the indexing behaviour seems fine.
hugh4life 3y ago
The past couple weeks there have been a few times I've been completely startled that google hadn't indexed a site. One of them is a forum that google used to completely index and now I have to use the forum search itself to find old posts.
retrocryptid 3y ago
My website used to be hosted by Google Sites. Then I moved it to something more "real" and deleted the Google sites version. The indexer still looks at their cache of the last version that was in sites from 5 or 7 years ago. So... the Google indexer doesn't work? SHOCKED! I AM SHOCKED TO FIND GOOGLE DOESN'T WORK!
friend_and_foe 3y ago
That's because google isn't a search engine. It's a content aggregator now.
kazinator 3y ago
> So how do you fix this? - Log into webmaster tools and find out what is going on? Maybe there are issues with the page. (I see Google doesn't call it webmaster tools any more.) - You can explicitly submit a page to be indexed. - To use the tools, you have to prove to Google that you own that domain: follow their instructions.
rldjbpin 3y ago
this makes me wonder if we will go back to the geocities model for hosting personal websites. perhaps it is already the case with .github.io and .substack.com. it has already been hard to discover websites hosted on their own domains, unless you find them organically (such as in HN).
hackerbeat 3y ago
Can confirm. Had to manually submit all URLs to Google to get them indexed. Once indexed, everything worked fine, like it used to back in the day. Also, this has been the case for every website launched over the past 12-18 months. In other words, this was already an issue before ChatGPT and the algo seems to be severely broken. High time for a thorough system check. Adsense reviews are just as messy. Another department in need of a big shakeup. I don't know what's going on there, but there's definitely lots of chaos behind the scenes.
scrumlord 3y ago
[dead]

599 points by c5karl 3y ago

An alternative web search architecture, is collaborative sharing the site we made or visit. I made a self hosted spider that only index sites that I've visited and search engine with "and" "or" logics. Thinking to make it a federated search engine by allow each user whitelist domains that are not sensitive to search with the peer. And users can follow/block other users explicitly to avoid spam. https://github.com/beenotung/personal-search-engine https://github.com/beenotung/personal-search-engine

FinnKuhn 3y ago

I tried this an I noticed that Bing doesn't seem to have this problem and has quite a few more pages indexed that Google.

KomoD 3y ago

Site is dead?

yabones 3y ago
Snapshot from a couple days ago: https://web.archive.org/web/20230531062024/https://natehoffelder.com/blog/google-no-longer-automatically-indexes-websites-wtf/ https://web.archive.org/web/20230531062024/https://natehoffe...
- Cockbrand 3y ago
  Or from the other popular archive site: https://archive.is/8bv2T https://archive.is/8bv2T
jmkni 3y ago
I thought you were referring to Google for a second lol
LorenDB 3y ago
Just the HN Hug O' Death at work again :)

BiteCode_dev 3y ago

I also noticed it takes way more time to index sites. When I moved bitecode.substack.com to https://bitecode.dev https://bitecode.dev, I submitted the sitemap to google and was surprised that it took almost a week to even show a single page as indexed. This used to be a couple of hours to a coupled of days. So something definitely changed, but as usual we can only speculate until a googler come in this thread.

gsatic 3y ago
Just more spam and scams than they can handle. 0.5% of all the shit on the Internet is real content. And that stat is from 2015. Their own fault for monetizing every imaginable search query. It has provided the incentive to ever spammer and scammer on the planet to mass produce copious amount fake interlinking pages massively larger than actual Internet.
- jodrellblank 3y ago
  Vernor Vinge was wrong about humanity's vile offspring being stock trading bots, and the sci-fi authors were wrong about Grey Goo being a nanobot scenario. We're awash in grey goo vile offspring right now, consuming resources at an increasing rate and turning everything bad - and it's information; SEO spam and content farms and LLM waffle. Pretty soon the only way to live will be an Encyclopaedia, text books, and cookbooks from before 2010.
  - gsatic 3y ago
    Its not a static world. Things are always changing. I have some faith Google(the ad tech part) will start crumbling. Chat GPT has given them a jolt. And there are more jolts to come. Not less. From the Regulators. From the advertisers who know most of the views they get are from bots or totally worthless. And that they don't need to be spending what they are spending.
    - pixl97 3y ago
      I'm not sure this fixes anything. GPT/LLM will be able to produce monumental amounts of reasonable sounding bullshit. Google may crumble and fail, but all we'll be left with is small islands of sanctuary in an ocean of bullshit.

pawelduda 3y ago

Maybe they ran out of disk space

deleted 3y ago
[deleted]
foobarbecue 3y ago
Sorry guys... I've been uploading all of every video I take to youtube... I should have known this would happen.
- ElFitz 3y ago
  That can’t beat those using YouTube as a low-cost backup platform. For those who missed it: https://gizmodo.com/backup-data-on-youtube-hack-white-noise-aka-isg-1850261527 https://gizmodo.com/backup-data-on-youtube-hack-white-noise-...
  - marcod 3y ago
    It's not very efficient?!? I am shocked!

philipov 3y ago

The site appears to have been hugged to death.

butler14 3y ago

Difficult to comment without URLs, there's too many potential causes Unfortunately "[being] a web designer since 2016" is generally not helpful for SEO But certainly setting up Google Search Console, submitting a sitemap, and 'fetching' your homepage on day 1 is good practice, especially for a brand new website

dskrepps 3y ago
If you're mass-producing low-quality sites you would do that on day 1 with each of them. So it likely doesn't weigh positively as much as we would hope.
user_666 3y ago
That is my blog post. I did that with the convention site, yes. It still wasn't getting indexed.
- BryanBigs 3y ago
  Did you build any links to the site? Links=Votes=Trust in Google's eyes....for better or worse. Even a tweet or Pinterest pin?
  - user_666 3y ago
    yep, all of that!
CTmystery 3y ago
I can also vouch that what the author is saying is true. I've started publishing my notes, and I found that after three weeks none of them were indexed (and yes I submit a sitemap through google webmaster tools). I have since started using the URL inspector tool and adding them one by one. That does work.

paxys 3y ago

[flagged]

gwbas1c 3y ago
Sometimes people like to experiment on their personal / side projects.
- hgsgm 3y ago
  Me too. But I don't claim to be an authority on web internals.
dom96 3y ago
The traffic isn't even that high. It boggles my mind any time a website cannot handle it.
- Ekaros 3y ago
  I always wondered. How did we manage to run anything lets say 20 years ago. We should have multiple times more memory, cpu, bandwidth available... Or are we just that many magnitudes more inefficient?
  - timw4mail 3y ago
    Yes, it is magnitudes more inefficient, with large JS payloads, slow APIs, and underpowered cloud hosts.
  - eterm 3y ago
    Websites often went down under load 20 years ago too but people were far more quick to think it was their own connection on the blink. Yes, sites were much better engineered too, with smaller and less rich and less dynamic payloads, but don't count out a change in expectations along with it.
- andersa 3y ago
  How much traffic could that be? Given the number of comments on this page, I can't imagine it being more than a few thousand clicks? So why would any site go down?

zolland 3y ago

[flagged]

KomoD 3y ago
That's just on archive.is

tyingq 3y ago

It seems obvious to me the importance of organic results has fallen over the years. There's plenty of queries where you have to scroll down a fair amount to even see them. Pushed down in favor of ads, various widgets, content sourced from Wikipedia and published on google's urls, etc. Things that sit below the fold get less overall money, time, and attention.

hackernewds 3y ago
With generative AI search results, soon you won't even be able to know whether your site was used for the result or not. Lots of no click queries, resulting in no traffic for the publisher
- userbinator 3y ago
  The much more worrying fact about AI is you won't even be able to know whether the information you're getting is true. I always scroll past the crap at the top to get to the actual site results.
  - danielheath 3y ago
    I haven’t been able to estimate that with confidence in a decade; SEO blogspam has seen to that.
    - nicbou 3y ago
      The content is true to the extent that a copywriter paraphrased another piece of content that is more authoritative. My work is getting paraphrased a lot, and they usually get the gist right, although they have zero clue of what they're talking about.
  - Karunamon 3y ago
    Given the web as it is today is infested with clickbait, "native content", clout-chasing, undisclosed sponsorship, and other such pathologies, I'm not convinced that the fear of AI making truth more rare is rational.

skilled 3y ago

Things are pretty bad with Google in this department. Over the last two years they have been constantly updating and changing their algorithm, there are now updates such as Core, Product Reviews, Helpful, Spam, Link Spam and God knows what else. And they are rotating them 24/7. Literally. You could be up 20% one day and down 40% the next and you are none the wiser as to why. If you are a small time publisher with no budget or no craftiness to attract links from other sites - you are pretty much doomed. I have a few sites that are in the 200k monthly visits range (from Google) and Google only fetches the homepage/feed every four hours or so, sometimes it takes longer than that. It’s a lot different from what it used to be.

jmclnx 3y ago

My site seemed to have come up fine. I wonder if it may be related to to the wayback machene "https://archive.org/web/ https://archive.org/web/" ? My site is on it has has been there for a while. I also have these on my WEB pages: <head> <META NAME="DESCRIPTION" CONTENT="MYTITLE"> <META NAME="KEYWORDS" CONTENT="COMMA SEPARATED LIST OF SIBJECTS"> <META NAME="AUTHOR" CONTENT="MYNAME"> <META NAME="RATING" CONTENT="General"> <META NAME="ROBOTS" CONTENT="index, follow"> <META NAME="REVISIT-AFTER" CONTENT="90 days"> </head> Maybe that will help out ? But I do notice google searches are not as good as years ago, and duckduckgo is close or maybe better than what google was.

metalrain 3y ago

Should Google index every website? I think answer is historically been yes, but more and more it's becoming no. Users of Google search want the best results, not all results. Customers of Google want their ad or site visible, not all sites.

Ekaros 3y ago
Funny thing is that very often I am not seeing the best result. Or even good result, but some stolen or SEO optimised junk... So to me it seems they are missing good content, not the bad.
- hospitalJail 3y ago
  I have a website that is #1 when it comes to the problem I'm solving. No one comes close. Its mostly due to, its a money saving thing for consumers, and that isnt really profitable. Its low hanging fruit, and I have the best website for it. Nothing really comes close, most alternative websites make mistakes in their advice because they are using feelings rather than millions of entries of data. Anyway, if you search specifically for my most popular metric, you will always get my website. If you google 'cheap X', you will get inferior websites. Even with SEO optimized, there are just bigger websites that are friendlier with google, linked by other websites, or it could be better SEO. Whatever the case, it makes me wonder what kind of websites I miss because I use google.
  - peter422 3y ago
    If the site is so good other people should naturally link to it over time and the situation will work itself out.
    - hospitalJail 3y ago
      It does, but the giants are just so much better at the simple phrase 'Save money on X'
deleted 3y ago
[deleted]

margarina72 3y ago

well, I can't say for everyone but I did notice that google stopped indexing a few sites I manage. No technical reason, no error, just says "Not indexed - reason: Discovered - currently not indexed" and that's most of the content of the site (nothing fancy mostly technical articles, and others notes) Unclear why this is happening.

gscott 3y ago
They won't index it until you have enough links coming into it.

flenserboy 3y ago

Google is an advertising & data-collection business, not a search company. Search is their on-ramp for attracting sources of data. If your website is not worth much on either side of the business, you'll be a low-priority target.

frouge 3y ago
Yes, Google is an ad-engine

eth0up 3y ago

I have a year old WP site that showed up on Google, but wasn't indexing all pages. I submitted the .xml sitemap via https://www.google.com/ping?sitemap="url" But that didn't seem fully effective. I then succumbed to uploading a verification file to the root directory (requires Google account) and resubmitted an inddx request. Within 12 hours a Google search of Site:"my url" yielded all pages. I'm terribly rusty with websites, hence my use of WordPress and willingness to taint my root directory with Google files. I do notice that exact, relevant queries in quotes still show no results for some content. Much to relearn.

franze 3y ago
was the sitemap.xml located in the root of the domain? https://www.google.com/ping?sitemap= https://www.google.com/ping?sitemap= only works if there is a previous root reference (i.e. via robots.txt or via GSC submit) of the sitemap.xml right issue, as from the original spec a sitemap can only submit URLs of the same or lower URL hierachy to Google. or there is a root reference to the sitemap.xml
- eth0up 3y ago
  I don't remember the initial loc. It's now in the root dir as mentioned. However, I'm still not pleased with the results, but perhaps it takes time. I'm sure backlink or two wouldn't hurt.

userbinator 3y ago

On the other side, I've also noticed it appears to be aggressively pruning its index in the past few years, so the fact that it's crawled your site doesn't mean it's necessarily searchable either. Another "bug" that seems to manifest quite often: if I search for a specific phrase or unique word on a page that I found in a SERP, so I know it's crawled that page, it often doesn't return that page either. Add to that the automatic CAPTCHA-hellban you get if you use "site:" in anything more than a tiny amount (and the one you still get if you search "too much"), and I realise that there's increasingly huge amounts of information out there on sites that Google may have crawled before and knows about, but doesn't want to show me for some reason. I remember it used to be much easier to find information about obscure topics even if it meant wading through dozens of pages of SEO spam; now it's nearly impossible for anything but the most vapid of queries.

nohuck13 3y ago
"Add to that the automatic CAPTCHA-hellban you get if you use "site:" in anything more than a tiny amount" Source? This would be worrisome.
- josephcsible 3y ago
  FWIW, I've personally experienced exactly that happening too.
- pteraspidomorph 3y ago
  I personally have been surprised to find myself CAPTCHA'd out of google search recently. No idea what's up with that. Regular commercial ISP, no VPNs.
- tyingq 3y ago
  Anecdotal, but this happens to me a lot, and not just with the "site:" operator. Generally using any of the advanced operators seems to set it off. Things like inurl:, intitle:, etc, trigger it also. Not every time, but after a few times. From a normal ISP connection, no VPN, even while logged into Google, etc.
- 3y ago

api 3y ago

I must once again plug Kagi.com. They're not perfect but are an actual search engine and I now use them as my primary. I am not affiliated with them.

arbitrage 3y ago
It appears you have to create an account before you see any search results. No thanks.
- allarm 3y ago
  Umm, subscription is actually the way Kagi monetises, instead of selling ads. This is exactly why you should consider it.
  - mrweasel 3y ago
    The complaint is still valid. I have subscriptions to a few sites, mostly news. I get why I need to login, otherwise how would they know that I've paid. It's still a pretty poor experience, I pay for a service, and now I need to fiddle with the settings in my browser so the site I pay for can remember my credentials. I get that part of the issue is that I keep a weird privacy focused configuration in my browsers and delete cookies when I close the browser, still it results in poor user experience, for a service that I pay for. I don't have any good suggestion on how to fix it, but it's is valid to complain about having to sign in.
    - efdee 3y ago
      It's a poor user experience because you break it on purpose. Whatever other way they can come up with to permanently identify you, you will block that in the name of privacy and keep complaining.
      - mrweasel 3y ago
        I don't really care about privacy from a company that already have my credit card information, and real name and address in most cases. I care about privacy from "the others", and the paid services is collateral damage. It does appear from another comment that Kagi actually thought about it and allows a token in the search URI, that's pretty neat.

hackdojo 3y ago

The site seems to be down (likely due to the site being on HN top page) As for actual indexing, we found the Indexing API to be the most reliable. https://console.cloud.google.com/apis/library/indexing.googleapis.com?project=pse-development https://console.cloud.google.com/apis/library/indexing.googl... The caveat is that you can only request Google up to 200 URLs to index. It's also technically not the most 'correct' way to have your pages indexed. (supposedly used for job page updates, etc)

voytec 3y ago

Hugged. Version from May 31: https://web.archive.org/web/20230531062024/https://natehoffelder.com/blog/google-no-longer-automatically-indexes-websites-wtf/ https://web.archive.org/web/20230531062024/https://natehoffe...

deleted 3y ago
[deleted]
moontear 3y ago
Thank you
- deleted 3y ago
  [deleted]
user_666 3y ago
Yeah, sorry about that - I didn't realize my story had been posted ehre until just a few minutes ago.
alpb 3y ago
How's it even hugged while being behind Cloudflare?
- luuurker 3y ago
  "cf-cache-status: DYNAMIC" The page isn't being cached. Cloudflare doesn't cache pages by default as that would create issues (logins, comments, admin sections, etc). They cache static files (images, css, js, etc), but that's it. You need to set it up yourself and have ways to bypass and purge the cache.
  - account42 3y ago
    > Cloudflare doesn't cache pages by default as that would create issues (logins, comments, admin sections, etc). They cache static files (images, css, js, etc), but that's it. You need to set it up yourself and have ways to bypass and purge the cache. Or they could just rely on Cache-Crontrol headers (and friends) instead of requiring CF-specific configuration.

kubanczyk 3y ago

Hugged to death. Appreciate the irony of reading it from Google's own cache. https://webcache.googleusercontent.com/search?q=cache:https%3A%2F%2Fnatehoffelder.com%2Fblog%2Fgoogle-no-longer-automatically-indexes-websites-wtf%2F https://webcache.googleusercontent.com/search?q=cache:https%...

digitcatphd 3y ago

I don't think this is a bug. After all, why waste resources indexing pages 2 .... n when nobody even goes there? If something accrues signals outside of Google naturally, then I am sure it will get indexed. On this note, I don't want to be another 'SEO is Dead' person, because it is never dead, it just changes, but I do suspect over the longer term, links will become less relevant and Google will take a more human-curated approach to ranking selection using LLMs.

Avamander 3y ago

They've been flooded with large amounts of new domains/sites hosting generated garbage content. I suspect this is just a barrier to slow it down until they figure out how to detect such spam better.

franze 3y ago

well, can't really comment without URL but one thing I found majorly annoying is who sticky "noindex" has become. used to be that you could put your staging on noindex, put it out there, do a lot of external testing tools (pages speed insights, mobile friendly tools, webpagetest.org, 10000 other tests that can easily be done on the open internet) and send the page around, and then on publishing day just remove the noindex, trigger recrawling and be indexed and visible in Google in no time. now it takes weeks to month for the noindex to vanish. even after google has crawled the pages again and again as visible in the logfiles, the pages stay noindex even though the information is long gone. google, the lazy monopoly strikes again.

user_666 3y ago
I've noticed this as well. I now use a different slug or domain for the development page(s), and change it when the page goes live.

gniv 3y ago

Indeed this very article doesn't come up in searches like [ddg site:natehoffelder.com] or [noindex site:natehoffelder.com]. And the article has been up since at least May 30. So yeah, looks like an outage at Google on the crawling side.

tensor 3y ago
When I go to the article I get some sort of cloudlflare "check" intermediate page. Could that be interfering with indexing?
- user_666 3y ago
  No, that was because I thought my site was under attack (this was before I learned the story was posted here).

bradley13 3y ago

Can confirm. While I don't do many websites anymore, I used to, and have since Google was the new kid on the block. I recently put up a new site, and...nothing. As the author said, I finally had to force each individual page to index. Harder to quantify, but it seems that search results are also worse than they used to be. It's possible that Google is doing some internal cost cutting...

danhak 3y ago

I'm experiencing this right now with my site https://www.dropspotter.com https://www.dropspotter.com I've done everything right according to Google, set up the search console, uploaded a site map, addressed all mobile usability issues. And yet only a tiny fraction of my content is being indexed. I'm a bit at the end of my rope here as I've poured a year into this project and getting a historically normal amount of search traffic may be the difference between this project being viable or not. The most frustrating part here is having zero visibility into what's going on.

soared 3y ago
Your site has very little “content” and is just product listings which looks like a million other garbage sites google sees. Great for ux to have the product listings users want, but the crawlers need some plain text to read to understand what you’re site is about. I’d add a paragraph, hero image image, cta/etc at the very top explaining what your site is. Additionally you need a menu at the top and footer at the bottom with links to additional content-only (not products) pages - IE an about us, where do we source data, etc. Even 2 blog posts would help a ton. Do not stuff them with keywords but be sure to use the words that are common in your niche of the industry so your site gets associated with industry sites. From googles eyes the difference between a scammy online ecom site and your site is hard to see! (Even if your site provides legitimate value to users). You can try posting a link to it on various digital Marketing subreddits (not the like “rate my website” ones) to see if you can get more feedback - I haven’t done that in years though. Edit - also didn’t realize clicking product links takes you to external sites. That’s a tough site for google to ever understand correctly since you have so little content and the best possible outcome of a user visiting your site is that they leave it. Maybe set it up to have each product link to your own page for it, maybe with price history from camelcamelcamel, links to the product in tiger sites, generic info about what site it’s listed on, or just crawl the description at the vendors site.
- danhak 3y ago
  Thank you for this extremely helpful response
3y ago

awesometech 3y ago

I think I'm going to try using DuckDuckGo for a while and compare, might even try Yahoo search.

nerdchum 3y ago
I found Brave search to be awesome. search.brave.com It's working on this idea called Brave Goggles where you can click on a goggle and it tweaks the algorithm to prioritize that type of content or specific sites get higher ranking It's pretty cool. All their results have been fantastic as well for me and it's supposed to be private as well. It's pretty much replaced google for me without even noticing.
- roter 3y ago
  You could do both: use DDG but preface search with !brave as needed.
user_666 3y ago
I mainly use DDG, and this is part of the reason why.
- efreak 3y ago
  The problem is that Google actually has support for quotes, even if it's not reliable. Try searching for short song lyrics or poetry, or pretty much any obscure quote on ddg and you can't find it. Putting individual words and phrases in quotes does nothing on ddg unless they suggest it for you...in which case it often still doesn't work

aa_is_op 3y ago

It will not index your website unless you register a Gmail address and then a Google Console account.

lispy4 3y ago
Also, Gmail is not playing nice with non-Gmail email accounts. If it's not coming from a large corporation, Gmail is filtering it.

nomadness 3y ago

kicked out of Google's index for no apparent reason, only to be thrown back in just as randomly. It's like a never-ending rollercoaster ride! The GSC comes to the rescue by helping to get important pages indexed fast, but guess what? You can only submit a measly 10 URLs per day! Ugh, talk about frustration!.. :-/

_joel 3y ago

It's a shame the website looks so broken (in ff)

TheCaptain4815 3y ago

Once again the black hats will come out on top. As someone in that industry, there's always a new trick or method to getting your sites indexed en mass before others. If you don't have time to find it, someone in a 2nd world country (probably India) will offer the service for relatively cheap at BlackHatWorld. On a side note on the SERP (search engine results page) of the foreseeable future, I don't accept the doom and gloom other SEOs have. Google has shown it will give credit to the top 3 links, and to be honest pretty visible credit. This will shift the target from "front page" to "top 3" sort of like Local Seo Map Packs. Will make difficult keywords more difficult to get clicks for, but long tail keywords could be easier to rank now as a #3 ranking could garner as many clicks as a #1 ranking.

danwee 3y ago

Google Search is in decline for the users who know how the internet works. For my mother, the internet is still Google (and would remain like that no matter what). For me, for some friends of mine, and for many of my colleagues who know more or less how things work on the web, Google Search is just in free fall: we use it as last resort, but as other search engines (or other tools, like ChatGPT) improve over time, Google Search would just disappear from our bookmarks. I know that many of you would say that it doesn't really matter what "hackers" think about Google Search, that all that matters is that the majority of the non-tech-savvy users still think Internet == Google. Well, let's talk again in 5 years.

okdood64 3y ago
I'm pretty sure you could find doom & gloom talk about Google on HN 5 years ago.
- pil0u 3y ago
  Only 7 years ago, the world was astonished with the release of AlphaGo. This was an algorithm dedicated to one very specific task of winning a game. The most recent game being played is language, a game that Google has played alone, until now. Doom & gloom talk about Google 5 years ago was probably unjustified and rightfully dismissed. Today, I'm not so sure about that.
- zarzavat 3y ago
  That’s because it’s been progressively getting worse for years. For example, when they launched Google+ they stopped supporting the + operator in favour of “quotes” and people complained about that. Of course now the “quotes” don’t work either. Google used to be amazing. If you remembered a set of words that were on a page, you could enter those words and it would find matching pages. It’s been broken for years and keeps getting more broken.
  - okdood64 3y ago
    Broken? https://www.google.com/search?q=%22may+have+been+less+attractive+to+candidates+heavily+focused+on+the+job+title%22&oq=%22may+have+been+less+attractive+to+candidates+heavily+focused+on+the+job+title https://www.google.com/search?q=%22may+have+been+less+attrac... I just took a random snippet from Gergely's blog and using quotes I found it. Look there's many issues with Google Search in 2023, but these nitpicks that aren't neccesarily true aren't productive to the conversation.

ahstilde 3y ago

If you're not in the top 5 results, there's no difference between not being indexed vs being indexed. If you want to be indexed, submit your pages to Google explicitly through a GA tag, a Google Search Console account, and a sitemap. If you're publishing many, many pages, you might see a benefit with the indexing API: https://developers.google.com/search/apis/indexing-api/v3/quickstart https://developers.google.com/search/apis/indexing-api/v3/qu... > I have been a web designer since 2016, and before that I was a blogger for 6 plus years. I have been deeply interested in Google SEO since 2010, and in all that time I’ve never heard of Google not indexing a site. Google crawls sites it assumes will have useful content. If it doesn't have that assumption for your site, you need to feed it.

ris58h 3y ago
> If you want to be indexed, submit your pages to Google explicitly through a GA tag, a Google Search Console account, and a sitemap. What if I don't want to put any Google-related stuff on my site? I have a GitHub pages site with a very unique topic and it's still not indexed by Google (the site is available at least for 8 month now). I've checked Bing and my site is on the 4-th place on the results page.
- dawnerd 3y ago
  Use a sitemap, that’s not google specific at all.
- ahstilde 3y ago
  > What if I don't want to put any Google-related stuff on my site? Then you're choosing to make it harder to be platformed by Google. Regardless, a GSC account and submitting a sitemap do not require you to serve any Google-related "stuff".
user_666 3y ago
That's my blog post. The convention site in question had a GA tag, and it was in search console, and I had submitted a sitemap. Search Console knee the site had 9 pages in the sitemap, and had only indexed the homepage. The other pages weren't rejected, declined or 404ed - they were simply ignored.

peter422 3y ago

There are an infinite number of URLs on the internet, Google obviously cannot crawl and index them all. They choose what to crawl and index based on mainly site reputation and inbound links, just like they used to 20 years ago. If they aren't crawling or indexing your site, then link to it from a high reputation place.

PaulHoule 3y ago

Google sees “putting your site online, getting it indexed, having it appear in search results, and receive traffic” to be competitive with “pay for advertising and premium placement.” The long term answer will be some kind of “Google Blue” where you have to pay for any placement, not just premium placement.

nforgerit 3y ago

So are we all now officially complaining about that it stinks where we've SEO-sh*tted for two decades? Good, then let's get rid of this ad network dystopia which never worked as advertised (they show me special offers for a frying pan, 2 weeks after I ordered one. Laughable ya folks). Replace it with great authentic inspirational content and people will throw their 2 bucks at it.

teddyh 3y ago
Too bad that the powers that be has decided that payments must be identified, so such a new internet will by necessity not be anonymous, but instead tracked and identified in every aspect.
- nforgerit 3y ago
  There's a difference between me deciding which payment method I use for service X and having service X decide into which tracking networks they put me in. Additionally, Mullvad showed how to use cash via regular post to pay 100% anonymously for service X.
  - teddyh 3y ago
    Oh, there are ways to do it nicely! But they won’t be used, and won’t be available.
executesorder66 3y ago
What's the point of censoring only one character from the word "shitted"? Both you and everyone that reads your comment know exactly what you are saying, so it's not like you are shielding anyone from anything. And neither is there anyone who is going to punish you for saying a naughty word on the internet. What are you afraid is going to happen if you write "shitted"?
- nforgerit 3y ago
  My mom's watching and soap is just not my kind of taste

ovulator 3y ago

I've noticed this for a side project I run. https://cfbpedia.com/ https://cfbpedia.com/ I upgraded it's server a while back and unknowingly borked it's SSL redirect. I corrected it and Google hasn't picked it up again since despite my fiddling in every way in their Console, still just "not indexed". If you search for it directly in Google you will find links to it, but never the site itself. I figured I was doing something wrong and didn't care enough to fix it, but it definitely seems like Google is making some sort of attempt at pruning.

ravenstine 3y ago

Maybe the concept of "web search" needs to be reimagined all together? Perhaps the idea of creating a master index of the content of multiple pages on nearly every site is the thing that is not sustainable. Instead, "search this site" needs to be made great again. Individual websites could manage their own search in a way that complies with a standard API that can be consumed by meta-search engines. Rather than indexing pages in the traditional way, meta-search engines instead use a heuristic or AI model to decide what sites are going to have the kind of information you are searching for, perform your query on theirs, and return the aggregated results to the user. The less the algorithm understands the significance or meaning of the query, the more generalized its approach can be. For instance, if it thinks that you're searching specifically for opinion-based content that will appear on blogs and forums, then it will target a federated search engine that indexes those things specifically. But if you are searching for information on making beer at home, it will know to target and weight the search engine on brewersfriend.com and homebrew.stackexchange.com. Although this sounds not that different to how search currently works today, remember that this idea is about having search become more federated and more standardized, and for meta-search to select federated indexes rather than own a god-index. A user of a meta-search can pick and choose what indexes they want available in their searches in case they find any of them to be either superior or particularly problematic, and the meta-search can optionally adjust its understanding of a particular user's queries. The way I see it, traditional search will continue to decline in part because it's not sustainable, but also in response to AI allowing them to become "answer engines". Although a lot of people do want an answer engine, this isn't for everyone. I think there will always be a market for people looking for content on specific webpages. Whatever that thing is that someday snatches that market away from The Google, if it's going to be successful, won't survive on the current concept of what "search" is.

codingdave 3y ago

I never thought they did it 'automatically'. My impression was that they always had to have found it via a link from another site, or that the site owner had to submit the site to Google. It isn't like Google is seeking out all registered domains and checking each one to see if a site suddenly appeared on it. So if someone registers a new domain, puts a site on it, doesn't link to it from anywhere else, doesn't submit to to Google... and yet expects the site to be found by Google, that is just not a reasonable expectation.

rightbyte 3y ago
I think they abuse Chrome or their 8.8.8.8 dns lookup somehow too. Sites certainly show up without links.

SL61 3y ago

I've seen this on a site that I built for a freelance client a couple years ago. At first they only indexed the homepage and some sporadic articles. Eventually one specific article was linked on an .edu website, and now that article is reliably indexed as well. If I manually request that they index a page, it always succeeds and shows up on Google within a few days, but the page gets pruned from the index within a month or two. The weird thing is that within the past couple years I've also seen unwanted indexing, e.g. low-spec staging servers getting wrecked by Google crawler traffic. I don't think the "no automatic indexing" treatment is standard for every site, so there must be something that triggers it, but I've spent a long time unsuccessfully trying to pin down the cause.

underlipton 3y ago

Have they fixed "pieces of me is actually a good song reddit" yet? (Google it.)

coldtea 3y ago

Perhaps they're not even trying anymore?

ltbarcly3 3y ago

Couldn't read, cloudflare 'verified my safety' and then hung for a couple minutes and I'm giving up.

sshine 3y ago
I'm in the same situation.

dbg31415 3y ago

Google has to know about the site before it can index it. Set up the sitemap, then link the sitemap in from Google's Search Console Tools, and install Google Analytics. This will help Google pick up that your site exists. Make sure your robots.txt file is configured to allow crawlers. Make sure your pages aren't inadvertently NOINDEX'd. SEO isn't as relevant as it used to be, but all this stuff should be part of your QA and pre-launch checklist. Get 90+ Lighthouse scores. https://developer.chrome.com/docs/lighthouse/overview/ https://developer.chrome.com/docs/lighthouse/overview/ Run your pages through WAVE or some other accessibility checker. https://wave.webaim.org/ https://wave.webaim.org/ Run your home page through Security Headers or some other front-end security audit. https://securityheaders.com/ https://securityheaders.com/ Set up uptime monitors on every page that gets more than 1% of your traffic... check page load speeds and HTTP response codes -- you never know when a WAF or some other system will get mucked up.

the_duke 3y ago

My personal guess: with ChatGPT and other locally runnable LLMs being commonplace now, Google just can't keep up anymore. There must be millions upon millions of SEO spam pages being created and updated every day. They lost the battle, at least for now. This also lines up with Google Search results becoming increasingly worse over the last few years, but the last months in particular.

zarzavat 3y ago
And who invented LLMs, transformers and modern AI? Google. There would be no ChatGPT without Google’s decade long campaign to create it.
Exuma 3y ago
Hmm, this is the best guess here IMO. This person did say it started occuring within the last year though. I just created a site a month ago with 2000+ pages and I did notice only 5 were indexed, which was quite annoying. I just suspected it was because I didn't have enough backlinks yet so it didn't want to waste time on it... interesting it may be what this original article is about. But yes, I 100% bet its because all new sites now have an enormous barrier with the sheer amount of fake content that can be generated.

boosteri 3y ago

Why does Nate Hoffelder need to review the security of my connection all of a sudden? It didnt seem to bother them an hour ago /s

user_666 3y ago
That is Cloudflare. So many people were clicking on the link from here that it inadvertently DDOS-ed my server. I had to turn on the "under attack" feature in Cloudflare just to get the other sites running again.

mo_42 3y ago

Google is part of a profit-maximizing company. Slowly, they are transitioning from showing relevant results and some ad in a separate box to results that maximize profit directly because they’re sponsored or whatever. The founders even explained it in one of their papers. Speculation: Crawling the web is expensive. So it makes sense that they decide for each crawled page if it's profitable to put that into the index. In the long run they might not crawl the web at all. People will just pay them to get into the index.

fps_doug 3y ago

As a random data point, I registered a new domain about a month ago and didn't do more than add a simple landing page yet, and it is properly indexed by Google.

mattbee 3y ago

Yup this confused me the other day - tried to search for an article I'd written for Dr. Dobbs Journal in 2001 on a very specific topic. No amount of keywords including my name, technical keywords etc. could find it. I assumed that meant it had been deleted from the web; the publication that had disappeared 10 years ago. Then saw I'd linked to it from my own site, and the link worked just fine :facepalm: It's weird to see Google forget the kind of content I used to go to them for in the first place.

giancarlostoro 3y ago

Google has been so weirdly different and off lately. I miss when I could type anything and get exactly what I was looking for. Those days died at least ten years ago. Sidenote: I wish Google would lower the ranking on websites who push for a mobile app and degrade your web browsing experience. There's no point in even getting them as a search result. If I wanted something from an app, I would have just gone on the app store.

ChatGTP 3y ago

Risk or not, I'm not sure why Sam Altman's views are really so important in this discussion. He is a CEO. Period.

WirelessGigabit 3y ago

The quality of searches have gone down so much. A lot of times these days you end up in these pages which have a page for every iteration (not the right word) of a sentence, pointed to their page full of 'answers'. Who is to blame? Google. For years they have pushed people to have slabs of text. Now I need to read through somebodies life story and how they stubbed their toe against a rock while hiking in the Sonoran foothills followed by a recipe for pulled pork tacos.

flumpcakes 3y ago

I think search results have become worse over time for three main reasons, firstly the amount of absolute drivel that's published on the web, made doubly worse with AI written drivel. It's honestly shocking. Secondly, the web has become more centralised and people use aggregators more than individual sites or services (facebook, reddit, etc.). Thirdly, the internet has become 'safer' in a lot of ways and seemingly the search engines scrub what is probably the majority of results from ever being returned. It used to be quite easy to find a pdf of a book on some open web server from a google search alone, now it's nearly impossible. And I don't think that's because people's security hygiene has improved.

koromak 3y ago

I don't understand how Google fails so miserably at doing its main job. I can't wait until I never have to use it again.

timbit42 3y ago
It's main job isn't search -- it's main job is advertising.

masswerk 3y ago

Another thing, I noticed: a weird prioritization of certain sources. E.g., I've a website that is now up for 24 years (there's even some content older than this, which has moved there), it has my name on it and my name is in the meta data, and – quite naturally – it used to be the top result, when you googled me. Recently, I discovered that the top results are now some university pages that are not maintained, or even have never been set up. Which really undermines the purpose of the site. (Notably, the site has quite a high number of reported Google search hits, has rather perfect Lighthouse scores, and SEO ratings are high enough that I get SEO related requests – which are happily ignored – on a rather regular basis. It scores about rank 60K in the "Majestic Million". And the site enjoys a few updates/fresh content a few times a year. So this is not a case of obscurity.)

lispy4 3y ago

Google has also begun aggressively filtering domains from sending email to Gmail users. This rolled out last November but they've ramped it up this past week to the point that many "indie" domains are 100% blocked from emailing Gmail accounts.

nubinetwork 3y ago
I haven't had issues sending emails to Gmail from my personal domains... but spamhaus/sorbs blocks Gmail on the inverse because people use it to send so much spam.

iza 3y ago

I've noticed my newer GitHub repos are not indexed by Google for weeks, while on Bing and DDG they show up very quickly.

renegat0x0 3y ago

That is why I wrote [1] for myself. It stores links in database, which I can query. Everything is later on exported, like in [2] and [3]. I can browse history, I can find useful data. I do not say it has replaced google for me. It is a nice addition that helped me gather data I encounter on the Internet. It is a link database, at first glance resembles Reddit clone, but my focus is on creating link database, not on providing social media experience cancer. Links: [1] https://github.com/rumca-js/Django-link-archive https://github.com/rumca-js/Django-link-archive [2] https://github.com/rumca-js/RSS-Link-Database https://github.com/rumca-js/RSS-Link-Database [3] https://github.com/rumca-js/RSS-Link-Database-2023 https://github.com/rumca-js/RSS-Link-Database-2023

jojohohanon 3y ago

Checking if site connection is secure And I left. Irony all exhausted

ChuckMcM 3y ago

My guess is that Google doesn't care (not that they ever did but now less than they did before). The simple math is "Google makes no money by indexing your website." The easiest way in the past to get reliably indexed by Google was to put Google ads on your web site. Then they have skin in the game so to speak. "Long tail" websites, those that are very niche, not especially linked outside of a few folks, have only ever been there to impress people using Google into believing they index everything (which they don't, and frankly haven't for at least the last 10 years). Over the years, as all of the ways that people who hosted a "search engine", exploited the "digital exhaust" of the people who used it, and sold that information for profit, have been revealed, well regulations and laws have been enacted to reduce or eliminate the egregious (and usually more profitable) ones. That and advertisers tired of paying for "engagement" and seeing few sales or conversions, the economics of running a search engine as this high margin, make money hand over fist, days are slowly winding down. That Bing has (for now at least), continued to index these sites, will just accelerate Google's decline. When I was helping to run Blekko, people would look at our curated results and be super impressed at how much better they were than Bing or Google results, but then they would search for their cousin's minecraft blog and wouldn't find it and lament that they couldn't possibly use it as their "daily driver" because if it didn't have their cousin's website in it, how could they know what they weren't seeing? Blekko tried really hard to make the argument that if you made a list of the minecraft blogs you followed as a Blekko user we'd index all of them, and if everyone did that for their favorite stuff then the index would fill up with good web sites and not be riddled with junk. But sadly we couldn't get them to internalize that and they weren't willing to create an account to have a better web experience. Perhaps we were just too early but still it is a weird thing.

marginalia_nu 3y ago
I think it's important to realize that search engines both bring websites to people and people to websites. The genius part of Google's position in the market is that they both sell ads, and then direct people to the places where the ads are sold. This is also a major contributor to their search engine spam problem; they can't well penalize ads without undermining their core business model. Although I don't think any search engine has ever indexed or will ever index everything. That doesn't really make sense. Realistically I think maybe 1% of the documents online are ever going to be a good search result for any query ever. Internet search is all about being judicious about what you index.

ffpip 3y ago

For some reason it refuses to index my site(s) until I link the site to my Google account in search console. I had organically placed a lot of links (on my GH, Linkedin profiles, no spam anywhere). It did not index even the links for over a month, despite me searching the exact domain multiple times. I guess doing it via search console just showed it the exact domains it has to crawl, making it's job easier

Melatonic 3y ago

Another reason I am glad I paid for Kagi hahahaha

zmix 3y ago

Each time I search for something, the first two, three pages, maybe even more, are just shops and affiliate link sites, that offer the thing I want information about. At least, if thing is a product. Not a single forum reference. I need to add the keyword "forum" in order to get such results. To me that is typical corporate degeneration and decline. Too big to fail, to trashy to be relevant any longer, instead, however, omnipresent.

Animats 3y ago

So this means you now must have a contractual relationship with Google to appear in Google search results? That raises major antitrust issues.

EMCymatics 3y ago

This should be a bigger deal

hannob 3y ago

I have recently started a new web project ( https://industrydecarbonization.com/ https://industrydecarbonization.com/ ) where I am relatively closely following how it does on search engines, and I can't say that I share those experiences. It took a few weeks until google noticed that it's a page with relevant content, but that's kinda expected. But once it did I feel Google is indexing my pages extremely fast, so fast that I have been wondering how they're actually doing this. I post new content on various social media sites, and my best guess is that google gets some of them as direct feeds that they check for interesting links. Google does not support indexnow, and as far as I know also no similar feature (except manually via the search console), so I'm not in any way directly submitting my content to Google.

deathanatos 3y ago

> and in all that time I’ve never heard of Google not indexing a site. […] Google used to just index every site whether you wanted them to or not Google can only index what they know of directly, or can indirectly discover e.g., by links. I have unindexed sites because, well, they're not linked to on the public Internet. I can pretty easily see a 2 month old website being outside that; the question is does the rest of the Internet know about it? (And, the public Internet. A shared link in a Discord channel also isn't visible to search engines…) Some of the examples contain statements that indicate Google could or should know of them, but not all of them.

paulrpotts 3y ago

I have a personal web site with almost 1500 pages containing blog posts, newsletters, reviews, and articles going back to the late eighties. I've been slowly consolidating all my writing on this site. For years I had kind of assume that if I build it, (eventually) it will be discoverable via organic search. I recently determined that I had to put a Google ID on it and a sitemap file. So I've found scripts, imperfect thought they are, to crawl everything and make a sitemap. Google has indexed some of it, but seems to be stalled out, with the bulk of the pages stuck in the "Discovered, currently not indexed" state, and now I'm not sure if some of the best content will _ever_ be searchable via Google. Really disappointing.

thedailymail 3y ago

It should be concerning for google that, 25 years since PageRank, search is once again ripe for disruption.

yoru-sulfur 3y ago

My website seems to have no indexing problems. I would be shocked if I had literally any other sites linking to mine (Small personal website), so at least from my perspective the indexing behaviour seems fine.

hugh4life 3y ago

The past couple weeks there have been a few times I've been completely startled that google hadn't indexed a site. One of them is a forum that google used to completely index and now I have to use the forum search itself to find old posts.

retrocryptid 3y ago

My website used to be hosted by Google Sites. Then I moved it to something more "real" and deleted the Google sites version. The indexer still looks at their cache of the last version that was in sites from 5 or 7 years ago. So... the Google indexer doesn't work? SHOCKED! I AM SHOCKED TO FIND GOOGLE DOESN'T WORK!

friend_and_foe 3y ago

That's because google isn't a search engine. It's a content aggregator now.

kazinator 3y ago

> So how do you fix this? - Log into webmaster tools and find out what is going on? Maybe there are issues with the page. (I see Google doesn't call it webmaster tools any more.) - You can explicitly submit a page to be indexed. - To use the tools, you have to prove to Google that you own that domain: follow their instructions.

rldjbpin 3y ago

this makes me wonder if we will go back to the geocities model for hosting personal websites. perhaps it is already the case with .github.io and .substack.com. it has already been hard to discover websites hosted on their own domains, unless you find them organically (such as in HN).

hackerbeat 3y ago

Can confirm. Had to manually submit all URLs to Google to get them indexed. Once indexed, everything worked fine, like it used to back in the day. Also, this has been the case for every website launched over the past 12-18 months. In other words, this was already an issue before ChatGPT and the algo seems to be severely broken. High time for a thorough system check. Adsense reviews are just as messy. Another department in need of a big shakeup. I don't know what's going on there, but there's definitely lots of chaos behind the scenes.

scrumlord 3y ago

[dead]