3 ms·
My guesswork based on the fact my URLs my crawler became aware of so many GitHub URLs I had to add special logic to exclude URLs that look like commit hashes:
by marginalia_nu 5y ago
My guesswork based on the fact my URLs my crawler became aware of so many GitHub URLs I had to add special logic to exclude URLs that look like commit hashes:
GitHub is absurdly large in terms of the number of documents since there's ostensibly a "document" per file and commit, it's likely the crawling budget Google affords GitHub simply runs out before they've crawled it all unless they implement special logic to prioritize issues and about-pages.