5 ms·
> If you want to run a few hundred parallel spider processes with full browser in each, it's neither easy nor cheap anymore. It takes a lot more resources than
by dsacco 9y ago
> If you want to run a few hundred parallel spider processes with full browser in each, it's neither easy nor cheap anymore. It takes a lot more resources than running it headless, plus adds a significant overhead to automate and control all that.
This is not true. I've seen it done in AWS for less than $2,000 monthly, and I can do it from my home for less than $500 per month (minus the costs of my workstation and networking gear, which you'd amortize over the expected lifetime of the project). I have a home server with 128GB of RAM and an i7-6900K, with 125 static IPs and a bunch of Ubiquiti networking gear. You don't even need that much memory or compute power, but I also use my workstation for other research projects. I use my own static IPs to parallelize without having to sacrifice latency or cede control to a shady proxy farm.
It's a pretty straightforward setup - each static IP is given its own route across the switches from the gateway, and the switches have link aggregated connections for 40GbE bandwidth. You have pub sub and queuing, and each scraping target is its own headless Chrome process. The requests to the target from each process are round robin sent across the available interfaces. Then you've got parsing and a local database.
I'm not a particularly invasive scraper (I make my User Agent deliberately obvious with an explicit way to opt out), but it's really not true that it's prohibitively expensive. This is a pretty cheap setup; I don't even profit from the work, I use it for personal research. If someone was actively selling valuable data, this would absolutely be worth it.
- toomuchtodo 9y agoWould you be willing to chat more about this one on one for research (non-commercial) purposes?
- dsacco 9y agoSure. I don't sell data but I'm happy to talk shop or help with interesting research projects.
- ivanhoe 9y agoIt's a matter of a personal perspective. If you're based in US and have a luxury of working with enterprise clients than sure. On the other hand, few years ago when I was doing data scrapping as a freelancer, price tag of $2K just for the infrastructure would be a huge show stopper for the most of my average clients back then. We were charging them way less than that.
- elorant 9y agoI'd love to read more details about your hardware setup. Care writing a blog post or something similar?
- nullpage 9y agoCurious about your home setup, what ISP are you using at home that lets you have essentially a /25 block of public IPs, let alone 40GbE of bandwidth? Especially if this is costing you $500/month.
- dsacco 9y agoI have Verizon Fios set up as a business account to my home. However that bandwidth is for SAN to workstation data processing and analysis. There's only half a gigabit of external internet bandwidth, but I have hundreds of terabytes of data (and high tens of gigabytes are downloaded per day) in local storage. With 128GB of RAM I can only load small amounts into memory for targeted analysis. Processing the rest of the data requires loading it directly from storage. To improve I/O performance I parallelize the transfer across link aggregated ethernet interfaces. Naturally that would cause disk reads to become a bottleneck; to take proper advantage of the network transfer speeds I hold all data in RAID 0 with 7200 RPM drives.
- cft 9y agoif your IPs are in a contiguous /25 , they can all be blocked with a single firewall rule. And detection is also easy. It only helps with rate limits. To avoid a block, you need a list of good socks5 proxies
- dsacco 9y ago> if your IPs are in a contiguous /25 They're not. > To avoid a block, you need a list of good socks5 proxies No; furthermore, that leaks your data to a third party and introduces significant latency.
- stordoff 9y agoCertainly curious about getting 40GbE bandwidth, but the IP addresses doesn't seem like much of a hurdle. I was looking at business broadband a couple of weeks ago, and getting 13 static IPs was only £5 extra a month. Going all the way up to a leased line included unlimited "subject to internet regulations".
- deleted 9y ago[deleted]