So? You can design a deterministic crawler for AI that - Respects robots.txt - Detects if something is likely a git forge and as a result - Clone the git rep

So? You can design a deterministic crawler for AI that - Respects robots.txt - Detects if something is likely a git forge and as a result - Clone the git repository instead of trying to crawl the forge None of this needs to be happening, and yet it does at an unprecedented scale.