3 ms·

Copying my comment from a previous discussion of ignoring robots.txt, below. I actually don’t care if someone ignores my robots.txt, as long as their crawler is

Copying my comment from a previous discussion of ignoring robots.txt, below. I actually don’t care if someone ignores my robots.txt, as long as their crawler is well run. But the smug attitude is annoying when so many crawlers are not. ———— We have a faceted search that creates billions of unique URLs by combinations of the facets. As such, we block all crawlers from it in robots.txt, which saves us AND them from a bunch of pointless indexing load. But a stealth bot has been crawling all these URLs for weeks. Thus wasting a shitload of our resources AND a shitload of their resources too. Whoever it is, they thought they were being so clever by ignoring our robots.txt. Instead they have been wasting money for weeks. Our block was there for a reason.

paulddraper 1y ago
Yes, this has been the traditional reason for robots.txt -- protects the bot as much as it does the site.
nonethewiser 1y ago
Wasting your money too right? I guess another angle on this is putting trust in people to comply with ROBOTS.txt. There is no guarantee so we should probably design with the assumption that our sites will be crawled however people want. Also im curious about your use case. >We have a faceted search that creates billions of unique URLs by combinations of the facets. Are we talking about a search that has filters like (to use ecommerce as an example), brand, price range, color, etc. And then all these combinations make up a URL (hence bilions)? How does a crawler discover these? They are just designed to detect all these filters and try all combinations? That doesn't really jive with my understanding with crawlers but otherwise IDK how it would be generating billions of unique URLs. I guess maybe they could also be included in sitemaps but I doubt that.
- xigoi 1y ago
  > How does a crawler discover these? They are just designed to detect all these filters and try all combinations? Presumably each of the facets is a link that adds it to the current query, so if you recursively follow links, you will end up with all combinations.
  - nonethewiser 1y ago
    But these links... where do they exist? In my experience they are produced from user INPUT. For example I click the sort button and sort by low to high and that adds a &sort=asc. Now I have a new url. After the input. But a crawler just visits the page right? Its not selecting from dropdowns, checkboxes, etc. So apparently all combinations are pregenerated somewhere? That seems weird to me and why I was asking about their use case.
freedomben 1y ago
I don't know anything about your specific use case, so take this with a grain of salt, but I've experienced this as well and digging in it is usually vulnerability scanning
chao- 1y ago
I have experienced the same situation with facet-like pages in the past. Links leading to (for example) the same product with a different color pre-selected on page load. Or within in a listing of a category products, a link might change from price descending to price ascending. All else equal, even the crawlers don't want to re-index the same page in eighty different ways if they could avoid it. They simply don't know better, and have decided to ignore our attempt to teach them (robots.txt). In the past, we've used this behavior as a signal to identify and block bad bots. These days, they will try again from 2000 separate residential IPs before they give up. But there was a long time where egregious duplicate page views of these faceted pages (against the advice of robots.txt) made detecting certain bad bots much easier.
rhet0rica 1y ago
I have two related stories. Googlebot has been playing a multiple-choice flash card game on my site for months—the page picks a random question and gives you five options to choose from. Each URL contains all of the state of the last click: the option you chose, the correct answer, and the five buttons. Naturally, Google wants to crawl all the buttons, meaning the search tree has a branch factor of five and search space of about 5000^7 possible pages. Adding a robots.txt entry failed to fix this—now the page checks the user agent and tells Googlebot specifically to fuck off with a 403. Weeks later, I'm still seeing occasional hits. Worst of all it's pretty heavy-duty—the flash cards are for learning words, and the page generator sometimes sprinkles in items that look similar to the correct answer (i.e., they have a low edit distance.) On the other hand there was a... thing crawling a search page on a separate site, but doing so in the most ass-brained way possible. Different IP addresses, all with fake user agents from real clients fetching search results for a database retrieval form with default options. (You really expect me to believe that someone on Symbian is fetching only page 6000 of all blog posts for the lowest user ID in the database?) The worst part about this one is that the URLs frequently had mangled query strings, like someone had tried to use substring functions to swap out the page number and gotten it wrong 30 times, resulting in Markov-like gibberish. The only way to get this foul customer to go away was to automatically ban any IP that used the search form incorrectly. So far I have banned 111,153 unique addresses. robots.txt wasn't adequate to stop this madness, but I can't say I miss Ahrefs or DotBot trying to gather valuable SEO information about my constructed languages.

paulddraper 1y ago

Yes, this has been the traditional reason for robots.txt -- protects the bot as much as it does the site.

nonethewiser 1y ago

Wasting your money too right? I guess another angle on this is putting trust in people to comply with ROBOTS.txt. There is no guarantee so we should probably design with the assumption that our sites will be crawled however people want. Also im curious about your use case. >We have a faceted search that creates billions of unique URLs by combinations of the facets. Are we talking about a search that has filters like (to use ecommerce as an example), brand, price range, color, etc. And then all these combinations make up a URL (hence bilions)? How does a crawler discover these? They are just designed to detect all these filters and try all combinations? That doesn't really jive with my understanding with crawlers but otherwise IDK how it would be generating billions of unique URLs. I guess maybe they could also be included in sitemaps but I doubt that.

xigoi 1y ago
> How does a crawler discover these? They are just designed to detect all these filters and try all combinations? Presumably each of the facets is a link that adds it to the current query, so if you recursively follow links, you will end up with all combinations.
- nonethewiser 1y ago
  But these links... where do they exist? In my experience they are produced from user INPUT. For example I click the sort button and sort by low to high and that adds a &sort=asc. Now I have a new url. After the input. But a crawler just visits the page right? Its not selecting from dropdowns, checkboxes, etc. So apparently all combinations are pregenerated somewhere? That seems weird to me and why I was asking about their use case.

freedomben 1y ago

I don't know anything about your specific use case, so take this with a grain of salt, but I've experienced this as well and digging in it is usually vulnerability scanning

chao- 1y ago

I have experienced the same situation with facet-like pages in the past. Links leading to (for example) the same product with a different color pre-selected on page load. Or within in a listing of a category products, a link might change from price descending to price ascending. All else equal, even the crawlers don't want to re-index the same page in eighty different ways if they could avoid it. They simply don't know better, and have decided to ignore our attempt to teach them (robots.txt). In the past, we've used this behavior as a signal to identify and block bad bots. These days, they will try again from 2000 separate residential IPs before they give up. But there was a long time where egregious duplicate page views of these faceted pages (against the advice of robots.txt) made detecting certain bad bots much easier.

rhet0rica 1y ago

I have two related stories. Googlebot has been playing a multiple-choice flash card game on my site for months—the page picks a random question and gives you five options to choose from. Each URL contains all of the state of the last click: the option you chose, the correct answer, and the five buttons. Naturally, Google wants to crawl all the buttons, meaning the search tree has a branch factor of five and search space of about 5000^7 possible pages. Adding a robots.txt entry failed to fix this—now the page checks the user agent and tells Googlebot specifically to fuck off with a 403. Weeks later, I'm still seeing occasional hits. Worst of all it's pretty heavy-duty—the flash cards are for learning words, and the page generator sometimes sprinkles in items that look similar to the correct answer (i.e., they have a low edit distance.) On the other hand there was a... thing crawling a search page on a separate site, but doing so in the most ass-brained way possible. Different IP addresses, all with fake user agents from real clients fetching search results for a database retrieval form with default options. (You really expect me to believe that someone on Symbian is fetching only page 6000 of all blog posts for the lowest user ID in the database?) The worst part about this one is that the URLs frequently had mangled query strings, like someone had tried to use substring functions to swap out the page number and gotten it wrong 30 times, resulting in Markov-like gibberish. The only way to get this foul customer to go away was to automatically ban any IP that used the search form incorrectly. So far I have banned 111,153 unique addresses. robots.txt wasn't adequate to stop this madness, but I can't say I miss Ahrefs or DotBot trying to gather valuable SEO information about my constructed languages.