3 ms·
One of the best use cases for "serverless" functions like AWS lambda is easily proxying around web crawling requests from the comfort of your codebase. To the d
by bschmidt1 2y ago
One of the best use cases for "serverless" functions like AWS lambda is easily proxying around web crawling requests from the comfort of your codebase. To the developer, it's just a function in a file, but it runs in isolation from a random IP address on each invocation - independent of app state. Like Puppeteer, one of Big Tech's little gifts to indie hackers.
- 0cf8612b2e1e 2y agoThat’s a technical solution to being a bad actor. I am not looking for ways of circumventing DDOS, more what do people consider a reasonable rate without putting undue burden on a host?
- bschmidt1 2y agoIt's part of the job at this point - it's public data, not like I'm stealing credit cards. Are you a more ethical developer because you "try to keep the hit rate low"? Can I pat myself on the back for only ever having 1 browser tab open at a time? Doesn't my minimalist take on browsing the web "keep my hit rate low" - I think I deserve a tax credit for helping the environment too, wdyt? Would it make you feel better if I said the pages I'm scraping are also hosted on AWS? That is - I'm effectively paying for the data by paying for lambdas. Or is it the poor hardware itself you worry for?