6 ms·
From the github issue, "it would be a fair bit of extra work for us to read robots.txt", so clearly tracking commit hashes would be even more work.
by callmeal 4y ago
From the github issue,
"it would be a fair bit of extra work for us to read robots.txt", so clearly tracking commit hashes would be even more work.