4 ms·
It seems to respect it as the majority of the requests are for the robots.txt.
by GaggiX 3y ago
It seems to respect it as the majority of the requests are for the robots.txt.
- flutas 3y agoHe says 3 million, and 1.8 million are for robots.txt So 1.2 million non robots.txt requests, when his robots.txt file is configured as follows # buzz off User-agent: GPTBot Disallow: / Theoretically if they were actually respecting robots.txt they wouldn't crawl any pages on the site. Which would also mean they wouldn't be following any links... aka not finding the N subdomains.
- swyx 3y agofor the 1.2 million are there other links he's not telling us about?
- flutas 3y agoI'm assuming those are homepage requests for the subdomains.
- otherme123 3y agoA lot of crawlers, if not all, have a policy like "if you disallow our robot, it might take a day or two before it notices". They surely follow the path "check if we have robots.txt that allows us to scan this site, if we don't get and store robots.txt, scan at least the root of the site and its links". There won't be a second scan, and they consider that they are respecting robots.txt. Kind of "better ask for forgiveness than for permission".
- jeremyjh 3y agoThat is indistinguishable from not respecting robots.txt. There is a robots.txt on the root the first time they ask for it, and they read the page and follow its links regardless.
- otherme123 3y agoI agree with you. I only stated how the crawlers seem to work, if you read their pages or try to block/slow down them it seems clear that they scan-first-respect-after. But somehow people understood that I approve that behaviour. For those bad crawlers, which I very much disapprove, "not respecting robots.txt" equals "don't even read robots.txt, or if I read it ignore it completely". For them, "respecting robots.txt" means "scan the page for potential links, and after that parse and respect robots.txt". Which I disapprove and don't condone.
- jeffnappi 3y agoHis site has a subdomain for every page, and the crawler is considering those each to be unique sites.
- sangnoir 3y agoThere are fewer than 10 links on each domain, how did GPTBot find out about the 1.8M unique sites? By crawling the sites it's not supposed to crawl, ignoring robots.txt. "disallow: /" doesn't mean "you may peek at the homepage to find outbound links that may have a different robots.txt"
- jameshart 3y agoOf course it’s considering them as unique sites. They are unique sites.
- vertis 3y agoExcept now it says # silly bing #User-agent: Amazonbot #Disallow: / # buzz off #User-agent: GPTBot #Disallow: / # Don't Allow everyone User-agent: * Disallow: /archive # slow down, dudes #Crawl-delay: 60 Which means he's changing it. The default for all other bots is to allow crawling.
- swatcoder 3y agoI'm not sure any publisher means for their robots.txt to be read as: "You're disallowed, but go head and slurp the content anyway so you can look for external links or any indication that maybe you are allowed to digest this material anyway, and then interpret that how you'd like. I trust you to know what's best and I'm sure you kind of get the gist of what I mean here."
- mminer237 3y agoHow would one know he is disallowed without reading each site?
- swatcoder 3y agoThe convention is that crawlers first read /robots.txt to see what they're encouraged to scrape and what they're not meant to, and then hopefully honor those directions. In this case, as in many, the disallow rules are intentionally meant to protect the signal quality and efficiency of the crawler.
- deleted 3y ago[deleted]