7 ms·

I have a single <a> element in my website's head, to a route banned by robots and the page is also marked by noindex meta tags and http headers. When something

by shakna 1y ago

I have a single <a> element in my website's head, to a route banned by robots and the page is also marked by noindex meta tags and http headers. When something grabs it, which AI crawlers regularly do, it feeds them the text of 1984, about a sentence per minute. Most crawlers stay on the line for about four hours.

dbalatero 1y ago
That's hilarious, can I steal the source for my own site?
- _moof 1y ago
  Only if you aren't a crawler.
  - E39M5S62 1y ago
    This is a long shot, but are you the same moof that ran the bot 'regurg' on EFnet in the late 90's / early 2000's for the BeOS community?
- anileated 1y ago
  That’s what LLM would say…