3 ms·
Exactly my observation as well. They devour absolutely everything, no exceptions. No matter how stupid it might be to digest a source code repository via HTTP.
by TonyTrapp 27d ago
Exactly my observation as well. They devour absolutely everything, no exceptions. No matter how stupid it might be to digest a source code repository via HTTP. They probably don't even recognize what's inside those pages and that there's an easier way to obtain the same result.
- diegocg 27d agoWhich, as the post notes, it's incredibly stupid. So much for artificial "intelligence"
- emsign 26d agoMakes me wonder how much garbage they actually collect across the web. That can't be good for the quality of the LLM.
- jeremyjh 26d agoThe crawlers are not AI. The crawlers are deterministic. They are collecting data to train AIs.
- jonhohle 26d agoBut gitweb is probably the second most used method of hosting a git repo and easily recognizable through heuristics. If it’s gitweb, fallback to git access and save everyone, including the crawler, time and resources.
- jeremyjh 26d agoUnderstood. The comment I replied to suggests the crawlers should figure this out on their own.
- TonyTrapp 26d agoNo, when I said this behaviour is stupid I did not imply that it needs to be solved with "intelligence". This class if problem is already solved by traditional crawlers, AI companies just actively chose to disregard any of that in their race to the top / bottom.
- TonyTrapp 26d agoSo? You can design a deterministic crawler for AI that - Respects robots.txt - Detects if something is likely a git forge and as a result - Clone the git repository instead of trying to crawl the forge None of this needs to be happening, and yet it does at an unprecedented scale.
- deleted 26d ago[deleted]