3 ms·

This is a mistake that many websites make, trying to block all robots, and the robots that serve their blog posts to users can't function anymore.

xnx 1y ago
Agree. If you don't want it out there, put it in your journal or require a login.
- reaperducer 1y ago
  Not every web site is a blog. Not every web site can be legally put behind a login.
  - dylan604 1y ago
    What kind of information legally cannot be put behind a login?
    - therein 1y ago
      Maybe he is talking about stuff you're required by law to disclose but you don't really want to be seen too much. Like code of conduct, terms of service, retractions or public apologies.
    - PeterStuer 1y ago
      Worst offenders I come across: official government information that needs to be public, placed behind Cloudflare, preventing even their M2M feeds (RSS, Atom, ...) to be accessed
      - dylan604 1y ago
        Is cloudflare providing some sort of login screen? How is that possible? Surely, you're not confusing a CAPTCHA with a login?
        PeterStuer 1y ago
        No logins, but browser fingerprinting, behavior tracking, straight blocking, captchas and other 'verification', even creating bogus honeypots and serving up unrelated generated spam instead of real page content.
        dylan604 1y ago
        None of that prevents users from getting to the data though so I’m wondering what your point is
        PeterStuer 1y ago
        The absurdity of e.g. the RSS feeds, specifically created to be consumed programatically, are blocked from being consumed programatically. Note that in 90% of the cases this was unitentional. Just Cloudflare enabeling lazy contracters with overzealous defaults. And as I said before, yes, you can write to get this corrected, and maybe in the next maitainance update next quarter it might get fixed. Until the next cheapest contracter wins the bid, wipes the config to minimize mai tainance, and upu can startall over again. The temaing 10% in case you wonder are malicious public bodies that want to pretend they do comply in making data public, but will do anything to make it as hard as possible to actually get to or consume yhat data. They love not just these blocking shenanigans, but will also hinder human browsing by putting in no translate pragmas, UX anti patterns, publish as poorly scanned PDFs, do not maintain canonical urls and shift things round constantly, remove things entirely "by mistake" even within the statutory period etc. Then there's very rarely (>.5%) activists IT employees on a true 'anti AI' crusade. This is where you encounter the most ridiculous homebrew 'solutions' that will often result in hurting also normal people accessing the site. This to them is just collateral in their rightous war.
bryanhogan 1y ago
Yes, there's often not much reason to block bots that abide by the rules. It just makes your site not show up on other search indexes and introduces problems for users. Malicious bots won't care about your robots.txt anyway.
happymellon 1y ago
Most bots don't serve the blog post to users.

xnx 1y ago

Agree. If you don't want it out there, put it in your journal or require a login.

reaperducer 1y ago
Not every web site is a blog. Not every web site can be legally put behind a login.
- dylan604 1y ago
  What kind of information legally cannot be put behind a login?
  - therein 1y ago
    Maybe he is talking about stuff you're required by law to disclose but you don't really want to be seen too much. Like code of conduct, terms of service, retractions or public apologies.
  - PeterStuer 1y ago
    Worst offenders I come across: official government information that needs to be public, placed behind Cloudflare, preventing even their M2M feeds (RSS, Atom, ...) to be accessed
    - dylan604 1y ago
      Is cloudflare providing some sort of login screen? How is that possible? Surely, you're not confusing a CAPTCHA with a login?
      - PeterStuer 1y ago
        No logins, but browser fingerprinting, behavior tracking, straight blocking, captchas and other 'verification', even creating bogus honeypots and serving up unrelated generated spam instead of real page content.
        dylan604 1y ago
        None of that prevents users from getting to the data though so I’m wondering what your point is
        PeterStuer 1y ago
        The absurdity of e.g. the RSS feeds, specifically created to be consumed programatically, are blocked from being consumed programatically. Note that in 90% of the cases this was unitentional. Just Cloudflare enabeling lazy contracters with overzealous defaults. And as I said before, yes, you can write to get this corrected, and maybe in the next maitainance update next quarter it might get fixed. Until the next cheapest contracter wins the bid, wipes the config to minimize mai tainance, and upu can startall over again. The temaing 10% in case you wonder are malicious public bodies that want to pretend they do comply in making data public, but will do anything to make it as hard as possible to actually get to or consume yhat data. They love not just these blocking shenanigans, but will also hinder human browsing by putting in no translate pragmas, UX anti patterns, publish as poorly scanned PDFs, do not maintain canonical urls and shift things round constantly, remove things entirely "by mistake" even within the statutory period etc. Then there's very rarely (>.5%) activists IT employees on a true 'anti AI' crusade. This is where you encounter the most ridiculous homebrew 'solutions' that will often result in hurting also normal people accessing the site. This to them is just collateral in their rightous war.

bryanhogan 1y ago

Yes, there's often not much reason to block bots that abide by the rules. It just makes your site not show up on other search indexes and introduces problems for users. Malicious bots won't care about your robots.txt anyway.

happymellon 1y ago

Most bots don't serve the blog post to users.