2 ms·

Myself and the person I was replying to ran our scrapers from our homes. One benefit being the IP address doesn’t get flagged. Curious why scraping from your ho

by city41 6y ago

Myself and the person I was replying to ran our scrapers from our homes. One benefit being the IP address doesn’t get flagged. Curious why scraping from your home network doesn’t work for you?

AlexITC 6y ago
Actually, that's what I'm doing. In the post I detailed that I run a service (https://cazadescuentos.net https://cazadescuentos.net) which is a browser extension to look for discounts. Sometimes, such service needs to look for the actual discounts, it queries the actual stores from the server that's on the cloud, but some stores doesn't let it go. So, I mounted a Pi at home that resolves only those problematic queries, I expose the Pi at home to my cloud server by using a SSH tunnel.
- city41 6y ago
  Yeah I saw that in your article. I’m wondering why bother with the cloud server? Why not scrape all stores from home?
  - AlexITC 6y ago
    The latency with the Pi is too high to handle all stores, I could easily do that if I was using a more powerful device.
    - city41 6y ago
      Ah I see, cool thanks for clarifying. That does make sense.