4 ms·

I run a number of news website scrapers for various projects. Originally I ran them all on AWS EC2 instances and racked up sizeable bills. Recently though, I pu

I run a number of news website scrapers for various projects. Originally I ran them all on AWS EC2 instances and racked up sizeable bills. Recently though, I pulled an unused Raspberry Pi from my desk drawer, installed them all onto the Pi, and now run them for near free from my desk! Granted I need to be a little bit more efficient with memory and processor cycles, but that’s a fun little challenge. I find if you respect the websites in terms of not hitting them multiple times a minute (or only once an hour which I do) then you’ll very seldom get banned.

AlexITC 6y ago
It's interesting how simple is to get small services out of the cloud now. I'm not that worried about websites banning the Pi as the traffic is really low, on concurrent calls it's unlikely to be a problem as I don't expect the Pi to query several websites in parallel. Certainly it's something worth implementing if I ever start getting more traffic. Thanks for sharing.
city41 6y ago
I did the same thing when I was running a scraper. I throttled the requests quite low to be respectful so it took a while to gather all the data, but it worked just fine. For this article, does the OP feel they need the proxy due to scraping rapidly?
- AlexITC 6y ago
  The problem isn't the speed, if it were, I would certainly use something more powerful than an old Pi. The actual problem is that some websites block IPs from known cloud providers, so, it doesn't matter which headers you use because what's blacklisted is the IP address.
  - city41 6y ago
    Myself and the person I was replying to ran our scrapers from our homes. One benefit being the IP address doesn’t get flagged. Curious why scraping from your home network doesn’t work for you?
    - AlexITC 6y ago
      Actually, that's what I'm doing. In the post I detailed that I run a service (https://cazadescuentos.net https://cazadescuentos.net) which is a browser extension to look for discounts. Sometimes, such service needs to look for the actual discounts, it queries the actual stores from the server that's on the cloud, but some stores doesn't let it go. So, I mounted a Pi at home that resolves only those problematic queries, I expose the Pi at home to my cloud server by using a SSH tunnel.
      - city41 6y ago
        Yeah I saw that in your article. I’m wondering why bother with the cloud server? Why not scrape all stores from home?
        AlexITC 6y ago
        The latency with the Pi is too high to handle all stores, I could easily do that if I was using a more powerful device.
        city41 6y ago
        Ah I see, cool thanks for clarifying. That does make sense.

AlexITC 6y ago

It's interesting how simple is to get small services out of the cloud now. I'm not that worried about websites banning the Pi as the traffic is really low, on concurrent calls it's unlikely to be a problem as I don't expect the Pi to query several websites in parallel. Certainly it's something worth implementing if I ever start getting more traffic. Thanks for sharing.

city41 6y ago

I did the same thing when I was running a scraper. I throttled the requests quite low to be respectful so it took a while to gather all the data, but it worked just fine. For this article, does the OP feel they need the proxy due to scraping rapidly?

AlexITC 6y ago
The problem isn't the speed, if it were, I would certainly use something more powerful than an old Pi. The actual problem is that some websites block IPs from known cloud providers, so, it doesn't matter which headers you use because what's blacklisted is the IP address.
- city41 6y ago
  Myself and the person I was replying to ran our scrapers from our homes. One benefit being the IP address doesn’t get flagged. Curious why scraping from your home network doesn’t work for you?
  - AlexITC 6y ago
    Actually, that's what I'm doing. In the post I detailed that I run a service (https://cazadescuentos.net https://cazadescuentos.net) which is a browser extension to look for discounts. Sometimes, such service needs to look for the actual discounts, it queries the actual stores from the server that's on the cloud, but some stores doesn't let it go. So, I mounted a Pi at home that resolves only those problematic queries, I expose the Pi at home to my cloud server by using a SSH tunnel.
    - city41 6y ago
      Yeah I saw that in your article. I’m wondering why bother with the cloud server? Why not scrape all stores from home?
      - AlexITC 6y ago
        The latency with the Pi is too high to handle all stores, I could easily do that if I was using a more powerful device.
        city41 6y ago
        Ah I see, cool thanks for clarifying. That does make sense.