4 ms·
What's the tech stack? How did you get around issues with your scrapers IP getting blocked?
by someuser54541 2y ago
What's the tech stack? How did you get around issues with your scrapers IP getting blocked?
- ICodeSometimes 2y agoHigh quality proxies, depending on the site i'll use anything from data center proxy up to residential :) Almost all requests goes through a proxy to be honest. You pay per GB transferred basically which works ok for me at the moment.
- someuser54541 2y agoDoes that wipe out your margins? Most proxies I'm aware of are relatively expensive and if you're charging per API request than there's not much margin to work with. What was the scraper written in? Python? Node? Go?
- ICodeSometimes 2y agoIt indeed does. My reasoning for creating this are the alternatives are WAY TOO EXPENSIVE (5-10x per api call). Since it's a one man show, i don't need much margin and i'm happy to just keep it alive since my friends and i use it for other projects that have more significant margin.