4 ms·

If your scraping program is fetching data at a similar rate and in a similar way to a browser, then yes, it shouldn't make any difference to the website owner.

by pdonis 5y ago

If your scraping program is fetching data at a similar rate and in a similar way to a browser, then yes, it shouldn't make any difference to the website owner. But most scraping programs are not like that. A web browser is fetching data for a human to read; that means it is not going to, for example, fetch a thousand pages from your website in just a few seconds using many parallel connections. But many scraping programs do things like that.

1vuio0pswjnm7 5y ago
This seems to suggest the only thing that should matter to the website owner is the manner of retrieving data, not the data being retrieved. If it is public data, that makes sense. I like to use sitemaps to download websites. I only use a single TCP connection and HTTP/1.1 pipelining. This can of take several successive requests depending on the number of URLs and the website's request limit per connection. For example, Akin Gump's sitemap has 9174 URLs and allows at least 2000 requests per connection. OTOH, I have downloaded sites with more URLs than that in a single connection. Pipelining is slower than parallel connections, more akin to downloading a large file. Some websites are faster than others. According to the original RFCs, this is the proper netiquette. Even though the practice is of using parallel connections is widespread, I have never been able to find an RFC that advocated that. Popular browsers today open dozens of connections. All in the name of advertising. I take the absence of a sitemap as an indication the website owner may have issues with automated data retrieval. For example, LinkedIn has no sitemap. However, most websites I encounter have sitemaps. Downloading websites without using parallel connections is not difficult. I do not need special software, only a small binary written in C to generate HTTP and a netcat. (Neither curl nor wget nor similar clients can do HTTP/1.1 pipelining.) I do not use a user-agent header. I have a filter I wrote for chunk transfer doecoding from stdin.

by pdonis 5y ago

1vuio0pswjnm7 5y ago

This seems to suggest the only thing that should matter to the website owner is the manner of retrieving data, not the data being retrieved. If it is public data, that makes sense. I like to use sitemaps to download websites. I only use a single TCP connection and HTTP/1.1 pipelining. This can of take several successive requests depending on the number of URLs and the website's request limit per connection. For example, Akin Gump's sitemap has 9174 URLs and allows at least 2000 requests per connection. OTOH, I have downloaded sites with more URLs than that in a single connection. Pipelining is slower than parallel connections, more akin to downloading a large file. Some websites are faster than others. According to the original RFCs, this is the proper netiquette. Even though the practice is of using parallel connections is widespread, I have never been able to find an RFC that advocated that. Popular browsers today open dozens of connections. All in the name of advertising. I take the absence of a sitemap as an indication the website owner may have issues with automated data retrieval. For example, LinkedIn has no sitemap. However, most websites I encounter have sitemaps. Downloading websites without using parallel connections is not difficult. I do not need special software, only a small binary written in C to generate HTTP and a netcat. (Neither curl nor wget nor similar clients can do HTTP/1.1 pipelining.) I do not use a user-agent header. I have a filter I wrote for chunk transfer doecoding from stdin.