3 ms·If they can't hit `/*/tree` is there a way to know the URLs of the files?by staplung 4y agoIf they can't hit `/*/tree` is there a way to know the URLs of the files?kadoban 4y agoSure, clone the git repo.utopcell 4y agoGitHub would not be happy with Google cloning all repos, and many of them at a high frequency, in order to circumvent a robots.txt restriction.kadoban 4y agoThey're clever people, they could just do partial updates (pull instead of clone). I doubt it'd be that much of a strain.pancrufty 4y agoDirect links from crawlable pages