3 ms·
Why do bots need to crawl so much? Any idea what kind of bots they are, is it mainly search engines?
by thinkloop 6y ago
Why do bots need to crawl so much? Any idea what kind of bots they are, is it mainly search engines?
- heimatau 6y ago> Why do bots need to crawl so much? The Earth has a (mostly) free society. Therefore, many programmers find a web crawler to be an advantageous endeavor. From search engines, to social media companies, to ad networks, to government agencies, to financial institutions, to students learning about programming, to nefarious actors, and even shadow versions of everything above. Bots are neat. One can do a lot with a freely accessible public database of information, only limit is one's creativity. (edited: for formatting)
- pilsetnieks 6y ago> only limit is one's creativity On the consumer side. There is a hard limit on bandwidth, though, on the provider side that can be exhausted by an overwhelming amount of bot traffic. In effect, that's what a DDoS is, even if unintentional.