12 ms·

I was wrong about robots.txt

171 points by EPendragon 1y ago

orionblastar 1y ago
This is a mistake that many websites make, trying to block all robots, and the robots that serve their blog posts to users can't function anymore.
- xnx 1y ago
  Agree. If you don't want it out there, put it in your journal or require a login.
  - reaperducer 1y ago
    Not every web site is a blog. Not every web site can be legally put behind a login.
    - dylan604 1y ago
      What kind of information legally cannot be put behind a login?
      - therein 1y ago
        Maybe he is talking about stuff you're required by law to disclose but you don't really want to be seen too much. Like code of conduct, terms of service, retractions or public apologies.
      - PeterStuer 1y ago
        Worst offenders I come across: official government information that needs to be public, placed behind Cloudflare, preventing even their M2M feeds (RSS, Atom, ...) to be accessed
        dylan604 1y ago
        Is cloudflare providing some sort of login screen? How is that possible? Surely, you're not confusing a CAPTCHA with a login?
        PeterStuer 1y ago
        No logins, but browser fingerprinting, behavior tracking, straight blocking, captchas and other 'verification', even creating bogus honeypots and serving up unrelated generated spam instead of real page content.
        dylan604 1y ago
        None of that prevents users from getting to the data though so I’m wondering what your point is
        PeterStuer 1y ago
        The absurdity of e.g. the RSS feeds, specifically created to be consumed programatically, are blocked from being consumed programatically. Note that in 90% of the cases this was unitentional. Just Cloudflare enabeling lazy contracters with overzealous defaults. And as I said before, yes, you can write to get this corrected, and maybe in the next maitainance update next quarter it might get fixed. Until the next cheapest contracter wins the bid, wipes the config to minimize mai tainance, and upu can startall over again. The temaing 10% in case you wonder are malicious public bodies that want to pretend they do comply in making data public, but will do anything to make it as hard as possible to actually get to or consume yhat data. They love not just these blocking shenanigans, but will also hinder human browsing by putting in no translate pragmas, UX anti patterns, publish as poorly scanned PDFs, do not maintain canonical urls and shift things round constantly, remove things entirely "by mistake" even within the statutory period etc. Then there's very rarely (>.5%) activists IT employees on a true 'anti AI' crusade. This is where you encounter the most ridiculous homebrew 'solutions' that will often result in hurting also normal people accessing the site. This to them is just collateral in their rightous war.
- bryanhogan 1y ago
  Yes, there's often not much reason to block bots that abide by the rules. It just makes your site not show up on other search indexes and introduces problems for users. Malicious bots won't care about your robots.txt anyway.
- happymellon 1y ago
  Most bots don't serve the blog post to users.
PaulKeeble 1y ago
The problem is the robots that do follow robots.txt its all the bots that don't. Robots.txt is largely irrelevant now they don't represent most of the traffic problem. They certainly don't represent the bots that are going to hammer your site without any regard, those bots don't follow robots.txt.
- _zagj 1y ago
  Not sure why you were downvoted. I have zero confidence that OpenAI, Anthropic, and the rest respect robots.txt however much they insist they do. It's also clear that they're laundering their traffic through residential ISP IP addresses to make detection harder. There are plenty of third-parties advertising the service, and farming it out affords the AI companies some degree of plausible deniability.
  - chneu 1y ago
    Nobody has any confidence in ai to not ddos. That's why there have been dozens of posts about how bandwidth is becoming an issue for many websites as bots continuously crawl their sites for new info. Wikipedia has backups for this reason. AI companies ignore the readily available backups and instead crawl every page hundreds of times a day. I think debian also recently spoke up about it.
  - EPendragon 1y ago
    One way to test if the model respects robots.txt could be to ask the model if it can scrape a given URL. One thing that it doesn't address though is the scrapping of the training data for the model. That area feels more like a wild west.
- zarzavat 1y ago
  That's what honeypots are for. Deny /honeypot in your robots.txt Add <a href="/honeypot" style="display:none" aria-hidden="true">ban me</a> to your index.html If an IP accesses that path, ban it.
  - deleted 1y ago
    [deleted]
s-mon 1y ago
Having worked on bot detection in the past. Some really simple old fashioned attacks happened by doing the opposite of what the robots.txt file says. While I doubt it does much today, that file really only matters to those that want to play by the rules which on the free web is not an awful lot of the web anymore I’m afraid.
- EPendragon 1y ago
  That was the first thing that I have learnt about the robots.txt file. Even RFC 9309 Robots Exclusion Protocol document: https://www.rfc-editor.org/rfc/rfc9309.html https://www.rfc-editor.org/rfc/rfc9309.html - mentions: > These rules are not a form of access authorization. Meaning that these are not enforced in any way. They cannot prevent you from accessing anything really. I think the only approach that could work in this scenario would be to find which companies disregard the robots.txt, and bring it to the attention of technical community. Practices like these could make the company look shady and untrustworthy if found out. That could be one way to keep them accountable, even though there is still no guarantee they will abide by it.
dumbfounder 1y ago
I created a search engine that crawled the web way back in 2003. I used a proper user agent that included my email address. I got SO many angry emails about my crawler, which played as nice as I was able to make it play. Which was pretty nice I believe. If it’s not Google people didn’t want it. That’s a good way to prevent anyone from ever competing with Google. It isn’t just about that preview for LinkedIn, it’s about making sure the web is accessible by everyone and everything that is trying to make its way. Sure, block the malicious ones. But don’t just assume that every bot is malicious by default.
- tomrod 1y ago
  > But don’t just assume that every bot is malicious by default. I'll bite. It seems like a poor strategy to trust by default.
  - ronsor 1y ago
    I'll bite harder. That's how the public Internet works. If you don't trust clients at all, serve them a login page instead of content.
    - __loam 1y ago
      It sucks that we're living in a landscape where bad actors take advantage of that way of doing things.
      - sltkr 1y ago
        The really bad actors are going to ignore robots.txt entirely. You might as well be nice to the crawlers that respect robots.txt.
        PeterStuer 1y ago
        Even if you want to play nice, robots.txt is a catch-22, as accessing it is taken as a signal you are a 'bot' by malconfigured anti-bot 'solutions'.
Falkon1313 1y ago
This is kinda amusing. robots.txt main purpose back in the day was curtailing penalties in the search engines when you got stuck maintaining a badly-built dynamic site that had tons of dynamic links and effectively got penalized for duplicate content. It was basically a way of saying "Hey search engines, these are the canonical URLs, ignore all the other ones with query parameters or whatever that give almost the same result." It could also help keep 'nice' crawlers from getting stuck crawling an infinite number of pages on those sites. Of course it never did anything for the 'bad' crawlers that would hammer your site! (And there were a lot of them, even back then.) That's what IP bans and such were for. You certainly wouldn't base it on something like User-Agent, which the user agent itself controlled! And you wouldn't expect the bad bots to play nicely just because you asked them. That's about as naive as the Do-Not-Track header, which was basically kindly asking companies whose entire business is tracking people to just not do that thing that they got paid for. Or the Evil Bit proposal, to suggest that malware should identify itself in the headers. "The Request for Comments recommended that the last remaining unused bit, the "Reserved Bit" in the IPv4 packet header, be used to indicate whether a packet had been sent with malicious intent, thus making computer security engineering an easy problem – simply ignore any messages with the evil bit set and trust the rest."
- pi_22by7 1y ago
  So it did the same work that a sitemap does? Interesting. Or maybe more like the opposite: robots.txt told bots what not to touch, while sitemaps point them to what should be indexed. I didn’t realize its original purpose was to manage duplicate content penalties though. That adds a lot of historical context to how we think about SEO controls today.
  - JimDabell 1y ago
    > I didn’t realize its original purpose was to manage duplicate content penalties though. That wasn’t its original purpose. It’s true that you didn’t want crawlers to read duplicate content, but it wasn’t because search engines penalised you for it – WWW search engines had only just been invented and they didn’t penalise duplicate content. It was mostly about stopping crawlers from unnecessarily consuming server resources. This is what the RFC from 1994 says: > In 1993 and 1994 there have been occasions where robots have visited WWW servers where they weren't welcome for various reasons. Sometimes these reasons were robot specific, e.g. certain robots swamped servers with rapid-fire requests, or retrieved the same files repeatedly. In other situations robots traversed parts of WWW servers that weren't suitable, e.g. very deep virtual trees, duplicated information, temporary information, or cgi-scripts with side-effects (such as voting). — https://www.robotstxt.org/orig.html https://www.robotstxt.org/orig.html
ceautery 1y ago
LinkedIn is by far the worst offender in post previews. The doctype tag must be all lowercase. The HTML document must be well-formed (the meta tags must be in an explicit <head> block, for example). You must have OG meta tags for url, title, type, and image. The url meta tag gets visited, even if it's the same address the inspector is already looking at. Fortunately, the post inspector helps you suss out what's missing in some cases, but c'mon, man, how much effort should I spend helping a social media site figure out how to render a preview? Once you get it right, and to quote my 13 year old: "We have arrived, father... but at what cost?"
- EPendragon 1y ago
  I had to discover the Post Inspector tool which was very helpful as it provided error messages. I didn't know that about the doctype tags. I must have had them right from the beginning. Didn't encounter those issues. It good to know though.
qwerty2000 1y ago
[flagged]
appleaday1 1y ago
isnt LinkedIn dead.
- kookamamie 1y ago
  One can dream.
- nxpnsv 1y ago
  It can still be more dead
- PeterStuer 1y ago
  I feel it is morphing into Twitter/Facebook/Instagram more each day. It used to be this ultrafake eternal job interview site, but people now seem uninhibited to go on wild political rants even there.
yodon 1y ago
What astounds me is there are no readily available libraries crawler authors can reach for to parse robots.txt and meta robots tags, to decide what is allowed, and to work through the arcane and poorly documented priorities between the two robots lists, including what to do when they disagree, which they often do. Yes, there's an ancient google reference parser in C++11 (which is undoubtedly handy for that one guy who is writing crawlers in C++), but not a lot for the much more prevalent Python and JavaScript crawler writers who just want to check if a path is ok or not. Even if bot writers WANT to be good, it's much harder than it should be, particularly when lots of the robots info isn't even in the robots.txt files, it's in the index.html meta tags.
- JimDabell 1y ago
  robots.txt support is built into the Python stdlib as urllib.robotparser: https://docs.python.org/3/library/urllib.robotparser.html https://docs.python.org/3/library/urllib.robotparser.html rel=nofollow is a bad name. It doesn’t actually forbid following the link and doesn’t serve the same purpose as robots.txt. The problem it was trying to solve was that spammers would add links to their site anywhere that they could, and this would be treated by Google as the page the links were on endorsing the page they linked to as relevant content. rel=nofollow basically means “we do not endorse this link”. The specification makes this more clear: > By adding rel="nofollow" to a hyperlink, a page indicates that the destination of that hyperlink should not be afforded any additional weight or ranking by user agents which perform link analysis upon web pages (e.g. search engines). > nofollow is a bad name […] does not mean the same as robots exclusion standards — https://microformats.org/wiki/rel-nofollow https://microformats.org/wiki/rel-nofollow
  - yodon 1y ago
    Thanks for this!
- codingminds 1y ago
  I don't see a reason why a good bot operator couldn't build a parser lib in a different language and put it on a public repo. Shouldn't be that hard if someone WANT to be good.
donohoe 1y ago
I try to stay away from negative takes here, so I’ll keep this as constructive as I can: It’s surprising to see the author frame what seems like a basic consequence of their actions as some kind of profound realization. I get that personal growth stories can be valuable, but this one reads more like a confession of obliviousness than a reflection with insight. And then they posted it here for attention.
- zem 1y ago
  it's mostly that they didn't think of the page preview fetcher as a "crawler", and did not intend for their robots.txt to block it. it may not be profound but it's at the least not a completely trivial realisation. and heck, an actual human-written blog post can okay improve the average quality of the web.
  - archievillain 1y ago
    The bots are called "crawlers" and "spiders", which to me evokes the image of tiny little things moving rapidly and mechanically from one place to another, leaving no niche unexplored. Spiders exploring a vast web. Objectively, "I give you one (1) URL and you traverse the link to it so you can get some metadata" still counts as crawling, but I think that's not how most people conceptualize the term. It'd be like telling someone "I spent part of the last year travelling." and when they ask you where you went, you tell them you commuted to-and-fro your workplace five times a week. That's technically travelling, although the other person would naturally expect you to talk about a vacation or a work trip or something to that effect.
    - JimDabell 1y ago
      > Objectively, "I give you one (1) URL and you traverse the link to it so you can get some metadata" still counts as crawling, but I think that's not how most people conceptualize the term. It’s definitely not crawling as robots.txt defines the term. : > WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages. — https://www.robotstxt.org/orig.html https://www.robotstxt.org/orig.html You will see that reflected in lots of software that respects robots.txt. For instance, if you fetch a URL with wget, then it won’t look at robots.txt. But if you mirror a site with wget, then it will fetch the initial URL, then it will find the links in that page, then before fetching subsequent pages it will fetch and check robots.txt.
kookamamie 1y ago
You shouldn't worry about LinkedIn, the cancer of the internet.
acosmism 1y ago
if you are hosting a house party that invites the entire world robots.txt is a neon sign to guide guests to where the beers are, who's cooking what kind of burgers and on what grill; rules of the house etc - you'll still have to secure your gold chains and laptop in a safe somewhere or decide to even keep them in the same house yourself
- throwawayffffas 1y ago
  Gold chains etc should be behind authentication. Robots txt is more like a warning sign that says the hedgemaze in the garden goes on forever, so probably stay out of it.
- EPendragon 1y ago
  A brilliant analogy. Robots doesn't provide access controls. Authentication and authorization do.
dwaite 1y ago
This doesn't seem like a new discovery at all - this is what news publications have been dealing with ever since they went online. You aren't going to get advertising without also providing value - be that money or information. Google has over 2 trillion in capitalization based primarily on the idea of charging people to get additional exposure, beyond what the information on their site otherwise would get.
- EPendragon 1y ago
  I believe that as search engines continue to move toward AI summaries and responses, it will reduce the traffic to the websites since most people will be ok accepting the answers that the AI gave them. My approach right now is to rely on social media traffic primarily where you can engage with the readers and build trust with the audience. I don't plan on using any advertising in the near future. While that might change, I am convinced that more intentional referral traffic will generate more intentional engagement.
NackerHughes 1y ago
This article could have been two lines. It takes some serious stretching of school-essay-writing muscles to inflate it to this many pages of waffle.
- EPendragon 1y ago
  I think a paragraph could have been enough to describe the issue. My goal with this post was to describe my personal experience with the problem, research, and the solution - the overall journey. I also wanted to write it in a manner that a non-technical person would be able to follow. Hence, being more explicit in my elaborations.
jarofgreen 1y ago
Hey OP, 1) You consider this about the Linkedin site but don't stop to think about other social networks. This is true about basically all of them. You may not post on Facebook, Bluesky, etc, but other people may like your links and post them there. I recently ran into this as it turns out the Facebook entries in https://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt also block the crawler FB uses for link previews. 2) From your first post, > But what about being closer to the top of the Google search results - you might ask? One, search engines crawling websites directly is only one variable in getting a higher search engine ranking. References from other websites will also factor into that. Kinda .... it's technically true that you can rank in Google if you block them in robots.txt but it's going to take a lot more work. Also your listing will look worse (last time I saw this there was no site description, but that was a few years back). If you care about Google SEO traffic you maybe want to let them on your site.
- EPendragon 1y ago
  Hey, @jarofgreen! Thank you for the feedback! 1) I only considered LinkedIn alone since I have been posting there and here on HN, and that's it. I figured I will let it play out until I need to allow for more bots to access it. Your suggestion of other people wanting to share the links to the blog is a very valid one that I haven't thought about. I might need to allow several other platforms. 2) With Google and other search engines I have seen a trend towards the AI summaries. It seems like this is the new meta for search engines. And with that I believe it will reduce the organic traffic from those engines to the websites. So, I do not particularly feel that this is a loss for me. I might eat my words in the future, but right now I think that social media and HN sharing is what will drive the most meaningful traffic to my blog. It is a word-of-mouth marketing, that I think is a lot more powerful than finding my blog in a Google search. I will definitely need to go back and do some more research on this topic to make sure that I'm not shooting myself in the foot with these decisions. Comments here have been very helpful in considering other opinions and options.
  - crowcroft 1y ago
    Point 2 is true to an extent, assuming you aren't monetizing your traffic though, wouldn't earning citations still be more valuable than not showing up in Google at all? You should also consider that a large proportion of search is purely navigational. What if someone is trying to find your blog and is typing 'evgenii pendragon'. AI summaries don't show for this kind of search. Currently I can see your site is still indexed (disallowing Google won't remove a page from the index if it's already been indexed) and so you show in search results, but because you block Google in robots.txt it can't crawl content and so shows an awkward 'No information is available for this page.' message. If you continue to block Google in robots.txt eventually the links to your site will disappear entirely. Even if you really don't want to let AI summarize your content, I would at least allow crawlers to your homepage.
alexey-salmin 1y ago
This reminds me of an old friend of mine who wrote long revelation posts on how he started using the "private" keyword in C++ after compiler helped him to find why a class member changed unexpectedly and how he no longer drives car with the clutch half-pressed because it burns the clutch.
jjcob 1y ago
I really think that most people should not use robots.txt If you don't want people to crawl your content, don't put it online. There are so many consequences of disallowing robots -- what about the Internet Archive for example?
- LightBug1 1y ago
  Nah, there's a middle way to that. I want to post online but I don't want random asshole driveby's involved.
- dspillett 1y ago
  > If you don't want people to crawl your content, don't put it online. I sometimes put things online for specific people to view/use, or for my own purposes. That gets an “all crawlers can do one” robots.txt and sometimes a script or two to try waste a little the time of those that don't play ball. It is online because I want it online, not for some random to hoover up. I consider robots.txt as a garden gate. I know it isn't secure, but likewise someone peering directly into my back bedroom window knows just as well that I don't want them there. I could put stuff like that behind authentication, but that is hassle for both me and the people who I want to be accessing the stuff. I usually use not-randomly-guessable URIs though sometimes that is inconvenient too, and anyway they do sometimes get “guessed”. I must have at least one friend-of-friend who has an active infestation which is reading their messages or browser history for things to probe because the traffic pattern isn't just preview generation, I've had known AI crawlers pass by some things repeatedly. TBH I don't really care that much, much at all in fact, I just don't like the presumption that my stuff is free for commercial exploitation.
franga2000 1y ago
The problem with robots.txt is the reliance on identity rather purpose of the bots. The author had blocked all bots because they wanted to get rid of AI scrapers. Then they wanted to unblock bots scraping for OpenGraph embeds so they unblocked...LinkedIn specifically. What if I post a link to their post on Twitter or any of the many Mastodon instances? Now they'd have to manually unblock all of their UA, which they obviously won't, so this creates an even bigger power advantage to the big companies. What we need is an ability to block "AI training" but allow "search indexing, opengraph, archival". And of course, we'd need a legal framework to actually enforce this, but that's an entirely different can of worms.
- BlackFly 1y ago
  This is just a problem of sharing information in band instead of out of band. The OpenGraph metadata is in band with the page content that doesn't need to be shared with OpenGraph bots. The way to separate the usage is to separate the content and metadata with some specific query using `content-type` or `HEAD` or something, then bots are free to fetch that (useless for AI bots) and you can freely forbid all bots from the actual content. Then you don't really need much of a legal framework.
  - EPendragon 1y ago
    I like the idea of using HEAD or OPTIONS methods and have all bots access that so that they get a high level idea of what's going on, without the access to actual content, if the owner decided to block it.
- zajio1am 1y ago
  Something like Common Crawl can be used for both search and AI training (and any other purpose, decided after crawl is done).
  - franga2000 1y ago
    Then such a crawler should mark itself with all purpose tags and thus be blocked in this scenario. Alternatively, it could make the request anyways and separate the crawled sites by permitted purpose in its output.
podgorniy 1y ago
What a way to pupmp up own online presence with near-to-nothing actions. I wish there were way less posts like this.
hiAndrewQuinn 1y ago
The "full solution" to this, of course, is micropayments. A bot which has to pay a tenth of a cent to you every time it visits one of your pages or something else the page 404s will quickly rack up a $10 bill crawling a whole 10,000 page site. If it tries to do that every day, or every hour, that's an excellent payday for you and a very compelling reason for almost all bots to blacklist your domain name. A human being who stops by to spend 20 minutes reading your blog once won't even notice they've spent 1.2 cents leafing through. This technology has existed for a while, and yet very few people have found it a good idea to wrap around. There is probably a good reason for that. The realistic solution is to probably just do some statistics and figure out who's getting on your nerves, and then ban them from your digital abode. Annoying, but people go a lot farther to protect their actual homes if they happen to live in high crime areas.
- Avnzx 1y ago
  Isn't this effectively what a tool like Anubis achieves?
klausa 1y ago
Weirdly, this is something that Apple actually gets right - the little „previews” you get when sharing links in iMessage get generated client-side; _by the sender_. There are good reasons why you’d not want to rely on clients providing this information when posting to LinkedIn (scams, phishing, etc); but it’s interesting to see an entirely different approach to the problem used here.
- pmontra 1y ago
  I came here to write that I expected that clients should generate previews after receiving a link inside a message. I also expected that somebody else would have already pointed that out and here we are. However I also understand that there are a number of reasons for a server to scrape the link. In no particular order: 1. scraping all the things, they might be useful sometimes in the future. 2. the page changes, goes 404, the client is reset and loses its db and can't rebuild the preview, but the client can rely on the server for that 3. it's faster for the client as the preview comes with the message and it does not have to issue some extra calls and parse a possibly slow page. Anyway you write that's the sender that generates the preview on iMessages so that's leaves point #1 and possibly the part of #2 about flaky internet connections: the server is in a better place to keep trying to generate the preview.
azangru 1y ago
> But what about being closer to the top of the Google search results - you might ask? One, search engines crawling websites directly is only one variable in getting a higher search engine ranking. References from other websites will also factor into that. As far as I remember from google search console, a disallow directive in robots.txt causes google not only to avoid crawling the page, but also to eventually remove the page from its index. It certainly shouldn't add any more pages to its index, external references or not.
- EPendragon 1y ago
  Thank you for that note! I didn't know that. It is something I will need to figure out: either I'm ok with not being in the search engine OR I will update the robots.txt. Currently I'm relying on the social media traffic and the word-of-mouth marketing.
naikrovek 1y ago
“Oh no my linked-in posts aren’t being put in front of enough people” It is amazing what people think is important these days.
darkwater 1y ago
This is a kind of scream test, even if self-inflicted. Scream tests are usually a good way to discover actual usage in complex (or not so complex) systems.
- EPendragon 1y ago
  It definitely was (and still is to a degree) self-inflicted :) It is fun to learn something when you discover an unintended consequence and then work backwards from it.
crowcroft 1y ago
Another common unintended consequence I've seen is conflating crawling and indexing with regards to robots.txt. If you make a new page and never want it to enter the Google search index, adding it to robots.txt is fine, Google will never crawl it and it will never enter the index. If you have an existing page that is currently indexed and want to remove it, adding that page to robots.txt is a BAD idea though. In the short term Google will continue to show the page in search results, but show it with no metadata (because it can't crawl it anymore). Even worse, Google won't notice up any noindex tags on the page, because robots.txt is blocking the page from being crawled! Eventually Google will get the hint and remove the page from the index, but it can be a very frustrating time waiting for that to happen.
- EPendragon 1y ago
  It is an interesting tidbit. I personally don't need Google to remove it from indexing. It is more of a "I don't care if they index it". I mostly care about the scrapping and not indexing. I do understand that these terms could be used interchangeably. In the past I might have conflated them.
- rafaelm 1y ago
  There are cases where Google might find a URL blocked in robots.txt (through external or internal links), and the page can still be indexed and show up in the search results, even if they can't crawl it. [1]. The only way to be sure that it will stay out of the results is to use a noindex tag. Which, as you mentioned, search engine bots need to "read" in the code. If the URL is blocked, the "noindex" cannot be read. [1] https://developers.google.com/search/docs/crawling-indexing/robots/intro#what-is-a-robots.txt-file-used-for https://developers.google.com/search/docs/crawling-indexing/... (refer to the red "Warning" section)
jrochkind1 1y ago
I think we are seeing the death of what was left of the open web, as people react to inconsiderate crawling for uses (AI) they are not sympathetic with by deciding trying to ban all automated access is the way to go. :( The result will be that giant corporations and those will bad intents will still find a way to access what they need, but small, hobby citizen and civil society efforts will be blocked out.
- EPendragon 1y ago
  I believe that this might give birth to new tools, protocols, and tech which will enable the next evolution of the Open Web into something akin to Protected Open Web. I very much dislike the invasive scrapping approaches. If something were to be done about it, it would result in a new way that clients interact with resources on the web.
lxgr 1y ago
robots.txt seems to be an irresistible attractor for some, most recently in a crusade against all kinds of GenAI. I get not wanting to have our data serve as training data, but I've also seen moderately large newspapers throwing literally all LLM bots in there, i.e. not only those that scrape training data, but also those that process users' search requests or even direct article fetches. The obvious, but possibly not expected, result was that this newspaper became effectively invisible to user searches in ChatGPT. Not sure if I'm an outlier here, but I personally click through to many of the sources ChatGPT provides, so they must be losing tons of traffic that way.
wltr 1y ago
A curious detail I noticed: the author’s account is a bit older than a month, yet he managed to publish four of his own articles (and nothing more, obviously). All of them (three) has zero comments and two of those three are dead. Yet, this post of his (posting his own work) gained traction. I believe for robots.txt topic rather than the article itself. That shows that even if you ignore all the rules of keeping a healthy community (not publish your self promotion only), eventually you’d get traction and nobody would care, I guess. Quick edit: my bad, wrong click brought me to the wrong location. So I made a bit wrong assumption. The author posted 4 extra posts alongside his own, so it’s not 100% of self-promotion, but 50%.
- EPendragon 1y ago
  To be honest, I also think that the discussion has become more interesting than what I wrote in the article. I have learned quite a few things for myself. And on top of that it has been great to hear some feedback in regards to some thoughts that I wrote. There are more articles available in my blog than I have shared here. I don't think that everything that I write is shareworthy on HN. There are some that I find to be more interesting. Those are the ones I end up sharing. Like you have noticed, I try to share other interesting resources that I find online too. Is there a ratio of self/others content that would help keeping a healthy community?
  - wltr 1y ago
    I don’t consider HN a healthy community, so I believe this approach of self-promoting the only sane thing to actually deal with this website. So, no real judgement from me. I just find it curious that apparently some mod banned you, but this time it was too late, since the discussion became somewhat interesting. If it was some other website, I’d say that if you post many of your own posts yourself many community members won’t be happy. I’d say go with not 50% but, say, at least 20% or even 10%. Again, my personal opinion of this website, it’s long beyond repair (if it ever was), so feel free to do whatever you want, unless some whims of mods ban you for no real reason one day.
    - 1y ago

171 points by EPendragon 1y ago

This is a mistake that many websites make, trying to block all robots, and the robots that serve their blog posts to users can't function anymore.

xnx 1y ago
Agree. If you don't want it out there, put it in your journal or require a login.
- reaperducer 1y ago
  Not every web site is a blog. Not every web site can be legally put behind a login.
  - dylan604 1y ago
    What kind of information legally cannot be put behind a login?
    - therein 1y ago
      Maybe he is talking about stuff you're required by law to disclose but you don't really want to be seen too much. Like code of conduct, terms of service, retractions or public apologies.
    - PeterStuer 1y ago
      Worst offenders I come across: official government information that needs to be public, placed behind Cloudflare, preventing even their M2M feeds (RSS, Atom, ...) to be accessed
      - dylan604 1y ago
        Is cloudflare providing some sort of login screen? How is that possible? Surely, you're not confusing a CAPTCHA with a login?
        PeterStuer 1y ago
        No logins, but browser fingerprinting, behavior tracking, straight blocking, captchas and other 'verification', even creating bogus honeypots and serving up unrelated generated spam instead of real page content.
        dylan604 1y ago
        None of that prevents users from getting to the data though so I’m wondering what your point is
        PeterStuer 1y ago
        The absurdity of e.g. the RSS feeds, specifically created to be consumed programatically, are blocked from being consumed programatically. Note that in 90% of the cases this was unitentional. Just Cloudflare enabeling lazy contracters with overzealous defaults. And as I said before, yes, you can write to get this corrected, and maybe in the next maitainance update next quarter it might get fixed. Until the next cheapest contracter wins the bid, wipes the config to minimize mai tainance, and upu can startall over again. The temaing 10% in case you wonder are malicious public bodies that want to pretend they do comply in making data public, but will do anything to make it as hard as possible to actually get to or consume yhat data. They love not just these blocking shenanigans, but will also hinder human browsing by putting in no translate pragmas, UX anti patterns, publish as poorly scanned PDFs, do not maintain canonical urls and shift things round constantly, remove things entirely "by mistake" even within the statutory period etc. Then there's very rarely (>.5%) activists IT employees on a true 'anti AI' crusade. This is where you encounter the most ridiculous homebrew 'solutions' that will often result in hurting also normal people accessing the site. This to them is just collateral in their rightous war.
bryanhogan 1y ago
Yes, there's often not much reason to block bots that abide by the rules. It just makes your site not show up on other search indexes and introduces problems for users. Malicious bots won't care about your robots.txt anyway.
happymellon 1y ago
Most bots don't serve the blog post to users.

PaulKeeble 1y ago

The problem is the robots that do follow robots.txt its all the bots that don't. Robots.txt is largely irrelevant now they don't represent most of the traffic problem. They certainly don't represent the bots that are going to hammer your site without any regard, those bots don't follow robots.txt.

_zagj 1y ago
Not sure why you were downvoted. I have zero confidence that OpenAI, Anthropic, and the rest respect robots.txt however much they insist they do. It's also clear that they're laundering their traffic through residential ISP IP addresses to make detection harder. There are plenty of third-parties advertising the service, and farming it out affords the AI companies some degree of plausible deniability.
- chneu 1y ago
  Nobody has any confidence in ai to not ddos. That's why there have been dozens of posts about how bandwidth is becoming an issue for many websites as bots continuously crawl their sites for new info. Wikipedia has backups for this reason. AI companies ignore the readily available backups and instead crawl every page hundreds of times a day. I think debian also recently spoke up about it.
- EPendragon 1y ago
  One way to test if the model respects robots.txt could be to ask the model if it can scrape a given URL. One thing that it doesn't address though is the scrapping of the training data for the model. That area feels more like a wild west.
zarzavat 1y ago
That's what honeypots are for. Deny /honeypot in your robots.txt Add <a href="/honeypot" style="display:none" aria-hidden="true">ban me</a> to your index.html If an IP accesses that path, ban it.
- deleted 1y ago
  [deleted]

s-mon 1y ago

Having worked on bot detection in the past. Some really simple old fashioned attacks happened by doing the opposite of what the robots.txt file says. While I doubt it does much today, that file really only matters to those that want to play by the rules which on the free web is not an awful lot of the web anymore I’m afraid.

EPendragon 1y ago
That was the first thing that I have learnt about the robots.txt file. Even RFC 9309 Robots Exclusion Protocol document: https://www.rfc-editor.org/rfc/rfc9309.html https://www.rfc-editor.org/rfc/rfc9309.html - mentions: > These rules are not a form of access authorization. Meaning that these are not enforced in any way. They cannot prevent you from accessing anything really. I think the only approach that could work in this scenario would be to find which companies disregard the robots.txt, and bring it to the attention of technical community. Practices like these could make the company look shady and untrustworthy if found out. That could be one way to keep them accountable, even though there is still no guarantee they will abide by it.

dumbfounder 1y ago

I created a search engine that crawled the web way back in 2003. I used a proper user agent that included my email address. I got SO many angry emails about my crawler, which played as nice as I was able to make it play. Which was pretty nice I believe. If it’s not Google people didn’t want it. That’s a good way to prevent anyone from ever competing with Google. It isn’t just about that preview for LinkedIn, it’s about making sure the web is accessible by everyone and everything that is trying to make its way. Sure, block the malicious ones. But don’t just assume that every bot is malicious by default.

tomrod 1y ago
> But don’t just assume that every bot is malicious by default. I'll bite. It seems like a poor strategy to trust by default.
- ronsor 1y ago
  I'll bite harder. That's how the public Internet works. If you don't trust clients at all, serve them a login page instead of content.
  - __loam 1y ago
    It sucks that we're living in a landscape where bad actors take advantage of that way of doing things.
    - sltkr 1y ago
      The really bad actors are going to ignore robots.txt entirely. You might as well be nice to the crawlers that respect robots.txt.
      - PeterStuer 1y ago
        Even if you want to play nice, robots.txt is a catch-22, as accessing it is taken as a signal you are a 'bot' by malconfigured anti-bot 'solutions'.

Falkon1313 1y ago

This is kinda amusing. robots.txt main purpose back in the day was curtailing penalties in the search engines when you got stuck maintaining a badly-built dynamic site that had tons of dynamic links and effectively got penalized for duplicate content. It was basically a way of saying "Hey search engines, these are the canonical URLs, ignore all the other ones with query parameters or whatever that give almost the same result." It could also help keep 'nice' crawlers from getting stuck crawling an infinite number of pages on those sites. Of course it never did anything for the 'bad' crawlers that would hammer your site! (And there were a lot of them, even back then.) That's what IP bans and such were for. You certainly wouldn't base it on something like User-Agent, which the user agent itself controlled! And you wouldn't expect the bad bots to play nicely just because you asked them. That's about as naive as the Do-Not-Track header, which was basically kindly asking companies whose entire business is tracking people to just not do that thing that they got paid for. Or the Evil Bit proposal, to suggest that malware should identify itself in the headers. "The Request for Comments recommended that the last remaining unused bit, the "Reserved Bit" in the IPv4 packet header, be used to indicate whether a packet had been sent with malicious intent, thus making computer security engineering an easy problem – simply ignore any messages with the evil bit set and trust the rest."

pi_22by7 1y ago
So it did the same work that a sitemap does? Interesting. Or maybe more like the opposite: robots.txt told bots what not to touch, while sitemaps point them to what should be indexed. I didn’t realize its original purpose was to manage duplicate content penalties though. That adds a lot of historical context to how we think about SEO controls today.
- JimDabell 1y ago
  > I didn’t realize its original purpose was to manage duplicate content penalties though. That wasn’t its original purpose. It’s true that you didn’t want crawlers to read duplicate content, but it wasn’t because search engines penalised you for it – WWW search engines had only just been invented and they didn’t penalise duplicate content. It was mostly about stopping crawlers from unnecessarily consuming server resources. This is what the RFC from 1994 says: > In 1993 and 1994 there have been occasions where robots have visited WWW servers where they weren't welcome for various reasons. Sometimes these reasons were robot specific, e.g. certain robots swamped servers with rapid-fire requests, or retrieved the same files repeatedly. In other situations robots traversed parts of WWW servers that weren't suitable, e.g. very deep virtual trees, duplicated information, temporary information, or cgi-scripts with side-effects (such as voting). — https://www.robotstxt.org/orig.html https://www.robotstxt.org/orig.html

ceautery 1y ago

LinkedIn is by far the worst offender in post previews. The doctype tag must be all lowercase. The HTML document must be well-formed (the meta tags must be in an explicit <head> block, for example). You must have OG meta tags for url, title, type, and image. The url meta tag gets visited, even if it's the same address the inspector is already looking at. Fortunately, the post inspector helps you suss out what's missing in some cases, but c'mon, man, how much effort should I spend helping a social media site figure out how to render a preview? Once you get it right, and to quote my 13 year old: "We have arrived, father... but at what cost?"

EPendragon 1y ago
I had to discover the Post Inspector tool which was very helpful as it provided error messages. I didn't know that about the doctype tags. I must have had them right from the beginning. Didn't encounter those issues. It good to know though.

qwerty2000 1y ago

[flagged]

appleaday1 1y ago

isnt LinkedIn dead.

kookamamie 1y ago
One can dream.
nxpnsv 1y ago
It can still be more dead
PeterStuer 1y ago
I feel it is morphing into Twitter/Facebook/Instagram more each day. It used to be this ultrafake eternal job interview site, but people now seem uninhibited to go on wild political rants even there.

yodon 1y ago

What astounds me is there are no readily available libraries crawler authors can reach for to parse robots.txt and meta robots tags, to decide what is allowed, and to work through the arcane and poorly documented priorities between the two robots lists, including what to do when they disagree, which they often do. Yes, there's an ancient google reference parser in C++11 (which is undoubtedly handy for that one guy who is writing crawlers in C++), but not a lot for the much more prevalent Python and JavaScript crawler writers who just want to check if a path is ok or not. Even if bot writers WANT to be good, it's much harder than it should be, particularly when lots of the robots info isn't even in the robots.txt files, it's in the index.html meta tags.

JimDabell 1y ago
robots.txt support is built into the Python stdlib as urllib.robotparser: https://docs.python.org/3/library/urllib.robotparser.html https://docs.python.org/3/library/urllib.robotparser.html rel=nofollow is a bad name. It doesn’t actually forbid following the link and doesn’t serve the same purpose as robots.txt. The problem it was trying to solve was that spammers would add links to their site anywhere that they could, and this would be treated by Google as the page the links were on endorsing the page they linked to as relevant content. rel=nofollow basically means “we do not endorse this link”. The specification makes this more clear: > By adding rel="nofollow" to a hyperlink, a page indicates that the destination of that hyperlink should not be afforded any additional weight or ranking by user agents which perform link analysis upon web pages (e.g. search engines). > nofollow is a bad name […] does not mean the same as robots exclusion standards — https://microformats.org/wiki/rel-nofollow https://microformats.org/wiki/rel-nofollow
- yodon 1y ago
  Thanks for this!
codingminds 1y ago
I don't see a reason why a good bot operator couldn't build a parser lib in a different language and put it on a public repo. Shouldn't be that hard if someone WANT to be good.

donohoe 1y ago

I try to stay away from negative takes here, so I’ll keep this as constructive as I can: It’s surprising to see the author frame what seems like a basic consequence of their actions as some kind of profound realization. I get that personal growth stories can be valuable, but this one reads more like a confession of obliviousness than a reflection with insight. And then they posted it here for attention.

zem 1y ago
it's mostly that they didn't think of the page preview fetcher as a "crawler", and did not intend for their robots.txt to block it. it may not be profound but it's at the least not a completely trivial realisation. and heck, an actual human-written blog post can okay improve the average quality of the web.
- archievillain 1y ago
  The bots are called "crawlers" and "spiders", which to me evokes the image of tiny little things moving rapidly and mechanically from one place to another, leaving no niche unexplored. Spiders exploring a vast web. Objectively, "I give you one (1) URL and you traverse the link to it so you can get some metadata" still counts as crawling, but I think that's not how most people conceptualize the term. It'd be like telling someone "I spent part of the last year travelling." and when they ask you where you went, you tell them you commuted to-and-fro your workplace five times a week. That's technically travelling, although the other person would naturally expect you to talk about a vacation or a work trip or something to that effect.
  - JimDabell 1y ago
    > Objectively, "I give you one (1) URL and you traverse the link to it so you can get some metadata" still counts as crawling, but I think that's not how most people conceptualize the term. It’s definitely not crawling as robots.txt defines the term. : > WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages. — https://www.robotstxt.org/orig.html https://www.robotstxt.org/orig.html You will see that reflected in lots of software that respects robots.txt. For instance, if you fetch a URL with wget, then it won’t look at robots.txt. But if you mirror a site with wget, then it will fetch the initial URL, then it will find the links in that page, then before fetching subsequent pages it will fetch and check robots.txt.

kookamamie 1y ago

You shouldn't worry about LinkedIn, the cancer of the internet.

acosmism 1y ago

if you are hosting a house party that invites the entire world robots.txt is a neon sign to guide guests to where the beers are, who's cooking what kind of burgers and on what grill; rules of the house etc - you'll still have to secure your gold chains and laptop in a safe somewhere or decide to even keep them in the same house yourself

throwawayffffas 1y ago
Gold chains etc should be behind authentication. Robots txt is more like a warning sign that says the hedgemaze in the garden goes on forever, so probably stay out of it.
EPendragon 1y ago
A brilliant analogy. Robots doesn't provide access controls. Authentication and authorization do.

dwaite 1y ago

This doesn't seem like a new discovery at all - this is what news publications have been dealing with ever since they went online. You aren't going to get advertising without also providing value - be that money or information. Google has over 2 trillion in capitalization based primarily on the idea of charging people to get additional exposure, beyond what the information on their site otherwise would get.

EPendragon 1y ago
I believe that as search engines continue to move toward AI summaries and responses, it will reduce the traffic to the websites since most people will be ok accepting the answers that the AI gave them. My approach right now is to rely on social media traffic primarily where you can engage with the readers and build trust with the audience. I don't plan on using any advertising in the near future. While that might change, I am convinced that more intentional referral traffic will generate more intentional engagement.

NackerHughes 1y ago

This article could have been two lines. It takes some serious stretching of school-essay-writing muscles to inflate it to this many pages of waffle.

EPendragon 1y ago
I think a paragraph could have been enough to describe the issue. My goal with this post was to describe my personal experience with the problem, research, and the solution - the overall journey. I also wanted to write it in a manner that a non-technical person would be able to follow. Hence, being more explicit in my elaborations.

jarofgreen 1y ago

Hey OP, 1) You consider this about the Linkedin site but don't stop to think about other social networks. This is true about basically all of them. You may not post on Facebook, Bluesky, etc, but other people may like your links and post them there. I recently ran into this as it turns out the Facebook entries in https://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt also block the crawler FB uses for link previews. 2) From your first post, > But what about being closer to the top of the Google search results - you might ask? One, search engines crawling websites directly is only one variable in getting a higher search engine ranking. References from other websites will also factor into that. Kinda .... it's technically true that you can rank in Google if you block them in robots.txt but it's going to take a lot more work. Also your listing will look worse (last time I saw this there was no site description, but that was a few years back). If you care about Google SEO traffic you maybe want to let them on your site.

EPendragon 1y ago
Hey, @jarofgreen! Thank you for the feedback! 1) I only considered LinkedIn alone since I have been posting there and here on HN, and that's it. I figured I will let it play out until I need to allow for more bots to access it. Your suggestion of other people wanting to share the links to the blog is a very valid one that I haven't thought about. I might need to allow several other platforms. 2) With Google and other search engines I have seen a trend towards the AI summaries. It seems like this is the new meta for search engines. And with that I believe it will reduce the organic traffic from those engines to the websites. So, I do not particularly feel that this is a loss for me. I might eat my words in the future, but right now I think that social media and HN sharing is what will drive the most meaningful traffic to my blog. It is a word-of-mouth marketing, that I think is a lot more powerful than finding my blog in a Google search. I will definitely need to go back and do some more research on this topic to make sure that I'm not shooting myself in the foot with these decisions. Comments here have been very helpful in considering other opinions and options.
- crowcroft 1y ago
  Point 2 is true to an extent, assuming you aren't monetizing your traffic though, wouldn't earning citations still be more valuable than not showing up in Google at all? You should also consider that a large proportion of search is purely navigational. What if someone is trying to find your blog and is typing 'evgenii pendragon'. AI summaries don't show for this kind of search. Currently I can see your site is still indexed (disallowing Google won't remove a page from the index if it's already been indexed) and so you show in search results, but because you block Google in robots.txt it can't crawl content and so shows an awkward 'No information is available for this page.' message. If you continue to block Google in robots.txt eventually the links to your site will disappear entirely. Even if you really don't want to let AI summarize your content, I would at least allow crawlers to your homepage.

alexey-salmin 1y ago

This reminds me of an old friend of mine who wrote long revelation posts on how he started using the "private" keyword in C++ after compiler helped him to find why a class member changed unexpectedly and how he no longer drives car with the clutch half-pressed because it burns the clutch.

jjcob 1y ago

I really think that most people should not use robots.txt If you don't want people to crawl your content, don't put it online. There are so many consequences of disallowing robots -- what about the Internet Archive for example?

LightBug1 1y ago
Nah, there's a middle way to that. I want to post online but I don't want random asshole driveby's involved.
dspillett 1y ago
> If you don't want people to crawl your content, don't put it online. I sometimes put things online for specific people to view/use, or for my own purposes. That gets an “all crawlers can do one” robots.txt and sometimes a script or two to try waste a little the time of those that don't play ball. It is online because I want it online, not for some random to hoover up. I consider robots.txt as a garden gate. I know it isn't secure, but likewise someone peering directly into my back bedroom window knows just as well that I don't want them there. I could put stuff like that behind authentication, but that is hassle for both me and the people who I want to be accessing the stuff. I usually use not-randomly-guessable URIs though sometimes that is inconvenient too, and anyway they do sometimes get “guessed”. I must have at least one friend-of-friend who has an active infestation which is reading their messages or browser history for things to probe because the traffic pattern isn't just preview generation, I've had known AI crawlers pass by some things repeatedly. TBH I don't really care that much, much at all in fact, I just don't like the presumption that my stuff is free for commercial exploitation.

franga2000 1y ago

The problem with robots.txt is the reliance on identity rather purpose of the bots. The author had blocked all bots because they wanted to get rid of AI scrapers. Then they wanted to unblock bots scraping for OpenGraph embeds so they unblocked...LinkedIn specifically. What if I post a link to their post on Twitter or any of the many Mastodon instances? Now they'd have to manually unblock all of their UA, which they obviously won't, so this creates an even bigger power advantage to the big companies. What we need is an ability to block "AI training" but allow "search indexing, opengraph, archival". And of course, we'd need a legal framework to actually enforce this, but that's an entirely different can of worms.

BlackFly 1y ago
This is just a problem of sharing information in band instead of out of band. The OpenGraph metadata is in band with the page content that doesn't need to be shared with OpenGraph bots. The way to separate the usage is to separate the content and metadata with some specific query using `content-type` or `HEAD` or something, then bots are free to fetch that (useless for AI bots) and you can freely forbid all bots from the actual content. Then you don't really need much of a legal framework.
- EPendragon 1y ago
  I like the idea of using HEAD or OPTIONS methods and have all bots access that so that they get a high level idea of what's going on, without the access to actual content, if the owner decided to block it.
zajio1am 1y ago
Something like Common Crawl can be used for both search and AI training (and any other purpose, decided after crawl is done).
- franga2000 1y ago
  Then such a crawler should mark itself with all purpose tags and thus be blocked in this scenario. Alternatively, it could make the request anyways and separate the crawled sites by permitted purpose in its output.

podgorniy 1y ago

What a way to pupmp up own online presence with near-to-nothing actions. I wish there were way less posts like this.

hiAndrewQuinn 1y ago

The "full solution" to this, of course, is micropayments. A bot which has to pay a tenth of a cent to you every time it visits one of your pages or something else the page 404s will quickly rack up a $10 bill crawling a whole 10,000 page site. If it tries to do that every day, or every hour, that's an excellent payday for you and a very compelling reason for almost all bots to blacklist your domain name. A human being who stops by to spend 20 minutes reading your blog once won't even notice they've spent 1.2 cents leafing through. This technology has existed for a while, and yet very few people have found it a good idea to wrap around. There is probably a good reason for that. The realistic solution is to probably just do some statistics and figure out who's getting on your nerves, and then ban them from your digital abode. Annoying, but people go a lot farther to protect their actual homes if they happen to live in high crime areas.

Avnzx 1y ago
Isn't this effectively what a tool like Anubis achieves?

klausa 1y ago

Weirdly, this is something that Apple actually gets right - the little „previews” you get when sharing links in iMessage get generated client-side; _by the sender_. There are good reasons why you’d not want to rely on clients providing this information when posting to LinkedIn (scams, phishing, etc); but it’s interesting to see an entirely different approach to the problem used here.

pmontra 1y ago
I came here to write that I expected that clients should generate previews after receiving a link inside a message. I also expected that somebody else would have already pointed that out and here we are. However I also understand that there are a number of reasons for a server to scrape the link. In no particular order: 1. scraping all the things, they might be useful sometimes in the future. 2. the page changes, goes 404, the client is reset and loses its db and can't rebuild the preview, but the client can rely on the server for that 3. it's faster for the client as the preview comes with the message and it does not have to issue some extra calls and parse a possibly slow page. Anyway you write that's the sender that generates the preview on iMessages so that's leaves point #1 and possibly the part of #2 about flaky internet connections: the server is in a better place to keep trying to generate the preview.

azangru 1y ago

> But what about being closer to the top of the Google search results - you might ask? One, search engines crawling websites directly is only one variable in getting a higher search engine ranking. References from other websites will also factor into that. As far as I remember from google search console, a disallow directive in robots.txt causes google not only to avoid crawling the page, but also to eventually remove the page from its index. It certainly shouldn't add any more pages to its index, external references or not.

EPendragon 1y ago
Thank you for that note! I didn't know that. It is something I will need to figure out: either I'm ok with not being in the search engine OR I will update the robots.txt. Currently I'm relying on the social media traffic and the word-of-mouth marketing.

naikrovek 1y ago

“Oh no my linked-in posts aren’t being put in front of enough people” It is amazing what people think is important these days.

darkwater 1y ago

This is a kind of scream test, even if self-inflicted. Scream tests are usually a good way to discover actual usage in complex (or not so complex) systems.

EPendragon 1y ago
It definitely was (and still is to a degree) self-inflicted :) It is fun to learn something when you discover an unintended consequence and then work backwards from it.

crowcroft 1y ago

Another common unintended consequence I've seen is conflating crawling and indexing with regards to robots.txt. If you make a new page and never want it to enter the Google search index, adding it to robots.txt is fine, Google will never crawl it and it will never enter the index. If you have an existing page that is currently indexed and want to remove it, adding that page to robots.txt is a BAD idea though. In the short term Google will continue to show the page in search results, but show it with no metadata (because it can't crawl it anymore). Even worse, Google won't notice up any noindex tags on the page, because robots.txt is blocking the page from being crawled! Eventually Google will get the hint and remove the page from the index, but it can be a very frustrating time waiting for that to happen.

EPendragon 1y ago
It is an interesting tidbit. I personally don't need Google to remove it from indexing. It is more of a "I don't care if they index it". I mostly care about the scrapping and not indexing. I do understand that these terms could be used interchangeably. In the past I might have conflated them.
rafaelm 1y ago
There are cases where Google might find a URL blocked in robots.txt (through external or internal links), and the page can still be indexed and show up in the search results, even if they can't crawl it. [1]. The only way to be sure that it will stay out of the results is to use a noindex tag. Which, as you mentioned, search engine bots need to "read" in the code. If the URL is blocked, the "noindex" cannot be read. [1] https://developers.google.com/search/docs/crawling-indexing/robots/intro#what-is-a-robots.txt-file-used-for https://developers.google.com/search/docs/crawling-indexing/... (refer to the red "Warning" section)

jrochkind1 1y ago

I think we are seeing the death of what was left of the open web, as people react to inconsiderate crawling for uses (AI) they are not sympathetic with by deciding trying to ban all automated access is the way to go. :( The result will be that giant corporations and those will bad intents will still find a way to access what they need, but small, hobby citizen and civil society efforts will be blocked out.

EPendragon 1y ago
I believe that this might give birth to new tools, protocols, and tech which will enable the next evolution of the Open Web into something akin to Protected Open Web. I very much dislike the invasive scrapping approaches. If something were to be done about it, it would result in a new way that clients interact with resources on the web.

lxgr 1y ago

robots.txt seems to be an irresistible attractor for some, most recently in a crusade against all kinds of GenAI. I get not wanting to have our data serve as training data, but I've also seen moderately large newspapers throwing literally all LLM bots in there, i.e. not only those that scrape training data, but also those that process users' search requests or even direct article fetches. The obvious, but possibly not expected, result was that this newspaper became effectively invisible to user searches in ChatGPT. Not sure if I'm an outlier here, but I personally click through to many of the sources ChatGPT provides, so they must be losing tons of traffic that way.

wltr 1y ago

A curious detail I noticed: the author’s account is a bit older than a month, yet he managed to publish four of his own articles (and nothing more, obviously). All of them (three) has zero comments and two of those three are dead. Yet, this post of his (posting his own work) gained traction. I believe for robots.txt topic rather than the article itself. That shows that even if you ignore all the rules of keeping a healthy community (not publish your self promotion only), eventually you’d get traction and nobody would care, I guess. Quick edit: my bad, wrong click brought me to the wrong location. So I made a bit wrong assumption. The author posted 4 extra posts alongside his own, so it’s not 100% of self-promotion, but 50%.

EPendragon 1y ago
To be honest, I also think that the discussion has become more interesting than what I wrote in the article. I have learned quite a few things for myself. And on top of that it has been great to hear some feedback in regards to some thoughts that I wrote. There are more articles available in my blog than I have shared here. I don't think that everything that I write is shareworthy on HN. There are some that I find to be more interesting. Those are the ones I end up sharing. Like you have noticed, I try to share other interesting resources that I find online too. Is there a ratio of self/others content that would help keeping a healthy community?
- wltr 1y ago
  I don’t consider HN a healthy community, so I believe this approach of self-promoting the only sane thing to actually deal with this website. So, no real judgement from me. I just find it curious that apparently some mod banned you, but this time it was too late, since the discussion became somewhat interesting. If it was some other website, I’d say that if you post many of your own posts yourself many community members won’t be happy. I’d say go with not 50% but, say, at least 20% or even 10%. Again, my personal opinion of this website, it’s long beyond repair (if it ever was), so feel free to do whatever you want, unless some whims of mods ban you for no real reason one day.
  - 1y ago