3 ms·

Wasting your money too right? I guess another angle on this is putting trust in people to comply with ROBOTS.txt. There is no guarantee so we should probably d

Wasting your money too right? I guess another angle on this is putting trust in people to comply with ROBOTS.txt. There is no guarantee so we should probably design with the assumption that our sites will be crawled however people want. Also im curious about your use case. >We have a faceted search that creates billions of unique URLs by combinations of the facets. Are we talking about a search that has filters like (to use ecommerce as an example), brand, price range, color, etc. And then all these combinations make up a URL (hence bilions)? How does a crawler discover these? They are just designed to detect all these filters and try all combinations? That doesn't really jive with my understanding with crawlers but otherwise IDK how it would be generating billions of unique URLs. I guess maybe they could also be included in sitemaps but I doubt that.

xigoi 1y ago
> How does a crawler discover these? They are just designed to detect all these filters and try all combinations? Presumably each of the facets is a link that adds it to the current query, so if you recursively follow links, you will end up with all combinations.
- nonethewiser 1y ago
  But these links... where do they exist? In my experience they are produced from user INPUT. For example I click the sort button and sort by low to high and that adds a &sort=asc. Now I have a new url. After the input. But a crawler just visits the page right? Its not selecting from dropdowns, checkboxes, etc. So apparently all combinations are pregenerated somewhere? That seems weird to me and why I was asking about their use case.