3 ms·
Even if it’s just http(s) requests that’s a lot of data to find & crawl. The bandwidth costs are probably insane.
by corentin88 7y ago
Even if it’s just http(s) requests that’s a lot of data to find & crawl. The bandwidth costs are probably insane.
- epoch_100 7y agoData ingress is usually free, which really cuts down on costs when scraping. If you can do everything in-memory, it's surprisingly cheap. The important bit is being respectful of robots.txt files and not overloading small sites with too many requests.
- JamesGreene 7y agoI have a background in scraping from prior projects over the last decade. Bandwidth is not a concern for projects like this at a lot of hosting/VPS providers.