3 ms·
I'm torn about their account. It's true that you could easily scrape 25k pages per day on a small VPS that costs less than the $50 Lambda costs they mentioned.
by heipei 6y ago
I'm torn about their account. It's true that you could easily scrape 25k pages per day on a small VPS that costs less than the $50 Lambda costs they mentioned. And in order to scrape from that VPS you wouldn't have to engineer this much with getting Chrome to run in Lambda, batching URLs, and you wouldn't worry about Lambda timeouts because you could run the whole scrape in one session more or less. So you could say that the engineering effort they spent building this was a waste of money. On the other hand, if they ever do need to scale up for whatever reason (information spread across more pages, or they need to scrape more services, or need multiple attempts per URL), all they have to do is push a button, at which point the upfront engineering effort will have paid off. Either way, their current Lambda costs are definitely eclipsed by the costs of paying for the residential proxy IPs. My two cents.