6 ms·

I have a side project that scrapes thousands and thousands of pages of a single website. So as not to piss them off (and so they don't try to block me), my scr

I have a side project that scrapes thousands and thousands of pages of a single website. So as not to piss them off (and so they don't try to block me), my script will take about 6 hours. Between each page fetch it sleeps for a small, random amount of time. It's been working like that for years.

Alifatisk 3y ago
Wish more people did like you!
- BlueTemplar 3y ago
  It's not just about "wishing", (D)DoSing is a punishable offence in many jurisdictions, and considering the damage taking down such a popular and essential website causes, I hope tIA will report them to law enforcement if it happens a third time.
bazmattaz 3y ago
Yes, this is the way. When I was purchasing a car last year I scraped a popular used car website for cars I liked to keep track of deals. I added a random sleep in between each page so the entire script took a few hours to run.
thangngoc89 3y ago
Yes. I did the same. My side project takes 4-5 days for the whole routine because there are random wait in between requests and only 2 active requests at any given moment.
perryprog 3y ago
What’s the benefit of waiting a random amount of time between requests?
- Lalabadie 3y ago
  Some zealous systems will infer a very regular request rate as coming from automated services and block them, no matter how gentle the rate.
  - coldpie 3y ago
    Is this speculation, or something that has actually been seen?
    - tastysandwich 3y ago
      I can't say how the website I'm scraping would respond if I just went full throttle. But it's also just a matter of courtesy anyway not to make ten thousands requests per second. Funny story - at work we once had a huge spike in requests from a single IP. We all crowded around, thinking it was some malicious hacker from France. How exciting - we're now interesting enough to warrant a DoS! Turns out another team in the company was just pulling all our data into Algolia to improve search. They were clearly not very courteous! So on the other end (building APIs) I certainly do pay attention to traffic and have Grafana alerts set up around it.
      - coldpie 3y ago
        I was asking about randomizing the interval between requests.
        pablosr11 3y ago
        tends to be around not repeating same patterns.
        cxr 3y ago
        Boy, this subthread has been exasperating to read.
    - otikik 3y ago
      I would block it, if I was the maintainer. As the linked post mentions, automated requests can "ramp up" at any moment, risking server stability. By preemptively blocking automated parsing (on a resource which primary usage is individual requests, not mass ones) I would avoid future problems for myself. Let them contact us via support if they really need an exception. In general I would rate limit by IP anything connected to the internet.
is_true 3y ago
Have you tried getting in touch to ask if they have an API?
that_guy_iain 3y ago
I build a system that scraped GitHub. Even though GitHub can clearly handle lots of traffic I still rate limited the hell out of it. The only time I've ever seen scrapers get banned is when go super fast. Unless it's LinkedIn and Instagram who guard their product aka data as much as possible.
- mdaniel 3y ago
  > I still rate limited the hell out of it Merely for your consideration, they actually do a great job of indicating in the response how many more requests per "window" the current authentication is allowed, and a header containing the epoch at which time the window will reset: https://docs.github.com/en/rest/overview/resources-in-the-rest-api?apiVersion=2022-11-28#rate-limit-headers https://docs.github.com/en/rest/overview/resources-in-the-re... I would suspect, all things being equal, that politely spaced requests are better than "as fast as computers can go" but I was just trying to point out that one need not bend oppressively over the other direction when the site is kind enough to tell the caller about the boundaries in a machine-readable format
  - that_guy_iain 3y ago
    That's on the API. For what I wanted it was do various HTTP requests to the website and do git clones.
galleywest200 3y ago
I recall back when I was less experienced I wrote a scraper for downloading recipes from Taste of Home and let Go firebomb them with as many goroutines as it could handle. I am sorry, admins! I won't do it next time.

Alifatisk 3y ago

Wish more people did like you!

BlueTemplar 3y ago
It's not just about "wishing", (D)DoSing is a punishable offence in many jurisdictions, and considering the damage taking down such a popular and essential website causes, I hope tIA will report them to law enforcement if it happens a third time.

bazmattaz 3y ago

Yes, this is the way. When I was purchasing a car last year I scraped a popular used car website for cars I liked to keep track of deals. I added a random sleep in between each page so the entire script took a few hours to run.

thangngoc89 3y ago

Yes. I did the same. My side project takes 4-5 days for the whole routine because there are random wait in between requests and only 2 active requests at any given moment.

perryprog 3y ago

What’s the benefit of waiting a random amount of time between requests?

Lalabadie 3y ago
Some zealous systems will infer a very regular request rate as coming from automated services and block them, no matter how gentle the rate.
- coldpie 3y ago
  Is this speculation, or something that has actually been seen?
  - tastysandwich 3y ago
    I can't say how the website I'm scraping would respond if I just went full throttle. But it's also just a matter of courtesy anyway not to make ten thousands requests per second. Funny story - at work we once had a huge spike in requests from a single IP. We all crowded around, thinking it was some malicious hacker from France. How exciting - we're now interesting enough to warrant a DoS! Turns out another team in the company was just pulling all our data into Algolia to improve search. They were clearly not very courteous! So on the other end (building APIs) I certainly do pay attention to traffic and have Grafana alerts set up around it.
    - coldpie 3y ago
      I was asking about randomizing the interval between requests.
      - pablosr11 3y ago
        tends to be around not repeating same patterns.
        cxr 3y ago
        Boy, this subthread has been exasperating to read.
  - otikik 3y ago
    I would block it, if I was the maintainer. As the linked post mentions, automated requests can "ramp up" at any moment, risking server stability. By preemptively blocking automated parsing (on a resource which primary usage is individual requests, not mass ones) I would avoid future problems for myself. Let them contact us via support if they really need an exception. In general I would rate limit by IP anything connected to the internet.

is_true 3y ago

Have you tried getting in touch to ask if they have an API?

that_guy_iain 3y ago

I build a system that scraped GitHub. Even though GitHub can clearly handle lots of traffic I still rate limited the hell out of it. The only time I've ever seen scrapers get banned is when go super fast. Unless it's LinkedIn and Instagram who guard their product aka data as much as possible.

mdaniel 3y ago
> I still rate limited the hell out of it Merely for your consideration, they actually do a great job of indicating in the response how many more requests per "window" the current authentication is allowed, and a header containing the epoch at which time the window will reset: https://docs.github.com/en/rest/overview/resources-in-the-rest-api?apiVersion=2022-11-28#rate-limit-headers https://docs.github.com/en/rest/overview/resources-in-the-re... I would suspect, all things being equal, that politely spaced requests are better than "as fast as computers can go" but I was just trying to point out that one need not bend oppressively over the other direction when the site is kind enough to tell the caller about the boundaries in a machine-readable format
- that_guy_iain 3y ago
  That's on the API. For what I wanted it was do various HTTP requests to the website and do git clones.

galleywest200 3y ago

I recall back when I was less experienced I wrote a scraper for downloading recipes from Taste of Home and let Go firebomb them with as many goroutines as it could handle. I am sorry, admins! I won't do it next time.