3 ms·
Understood. The comment I replied to suggests the crawlers should figure this out on their own.
by jeremyjh 2mo ago
Understood. The comment I replied to suggests the crawlers should figure this out on their own.
- TonyTrapp 2mo agoNo, when I said this behaviour is stupid I did not imply that it needs to be solved with "intelligence". This class if problem is already solved by traditional crawlers, AI companies just actively chose to disregard any of that in their race to the top / bottom.