4 ms·
Google docs are pretty clear (https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec https://developers.google.com/crawling/docs/robots-txt/robo
by WmWsjA6B29B4nfk 9mo ago
Google docs are pretty clear (https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec https://developers.google.com/crawling/docs/robots-txt/robot...):
> Google's crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn't exist. This means that Google assumes that there are no crawl restrictions.
This is a better source than a random SEO dude with a channel full of AI-generated videos.
- marginalia_nu 9mo agoNot entirely unlikely this is just a bug on Google's end. It's fairly common for there to be a very long and circuitous route between cause and effect in search, so a bug like this can sometimes be difficult to identify until people start making blog posts about it.
- efilife 9mo agoIt seems that this is not happening and even the guy who wrote the article mentions it: > I don't have a robots.txt right now. It hasn't been there in a long time. Google still shows two results when I search for files on my site though: The source that he links to is another indian spam channel we've seen a thousand times on YouTube
- xnx 9mo agoIt does seem unlikely that Google would have a big in basic behavior of its crawler after ~27 years.
- deleted 9mo ago[deleted]
- deleted 9mo ago[deleted]
- conradfr 9mo agoGoogle Adsense docs says that ads.txt is not mandatory and yet I remember having no ads displayed on my website until I added one.
- jamesfinlayson 9mo agoYeah I thought I got a notification saying to add it for an existing site but it still seemed optional the last time I created a new site?
- edwinjm 9mo agoIndeed. “Unreachable” is very different than “not found”.
- khana 9mo ago[dead]