4 ms·

All the (AWS) technologies used are totally unnecessary. SQS/DynamoDB/Lambda. I can buy a laptop in walmart for $500 and i can do all the scrapping in starbucks

All the (AWS) technologies used are totally unnecessary. SQS/DynamoDB/Lambda. I can buy a laptop in walmart for $500 and i can do all the scrapping in starbucks wifi.

rblatz 6y ago
Right, it seems like they overbuilt this hacky solution. You are scraping, eventually you just need to subscribe to the data. Why invest that much effort into a temporary solution.
deleted 6y ago
[deleted]
nunez 6y ago
Lambda is needed to get rotating IPs and scale while avoiding browser fingerprinting. SQS takes the results of those scrapes and puts them into a database, DynamoDB. It's a straightforward web scraping pipeline.
- atesti 6y ago
  No, the access google over residental proxy servers provided by packetstream.io
- toddh 6y ago
  Lambda isn't enough. You'll get blocked in a heartbeat. You still need a proxy service.
sdinsn 6y ago
Of course its unnecessary. The point is that you can do it in the cloud, instead of on a laptop at starbucks...