4 ms·
> Honestly though the big issue is crawling. Not only is it a bandwidth monster many sites are hostile to non google bots, along with cdn’s and cloudflare. I d
by marginalia_nu 5y ago
> Honestly though the big issue is crawling. Not only is it a bandwidth monster many sites are hostile to non google bots, along with cdn’s and cloudflare.
I do my own crawling and don't agree with any of these statements. Bandwidth is not a bottleneck, and blocking is mostly only a problem if your bot is too aggressive.
- boyter 5y agoCould be due to the long form content you index. I found those sort of sites tend to have less reluctance on 3rd parties. Also possible you are more talented than I with your crawler writing.