I've been building a webcrawler that casts a pretty wide net and I've yet to run into it. I'm not just talking about API endpoints and stuff - I've never run in

I've been building a webcrawler that casts a pretty wide net and I've yet to run into it. I'm not just talking about API endpoints and stuff - I've never run into this doing general crawls of the web.