3 ms·
I would definitely pick up the phone and ask but you have to admit there's always the danger of being said a very firm "NO!" and some people might actually inve
by pdimitar 8y ago
I would definitely pick up the phone and ask but you have to admit there's always the danger of being said a very firm "NO!" and some people might actually investigate who are you and take active measures to block any traffic coming from you (which is not very hard if you give them your email during the call).
So I would think most scraping services assume they will be refused when doing such requests so they never bother.
- jacquesm 8y agoIf you are anticipating a 'no' then you definitely should ask for permission. The only valid reason for starting a large scraping operation is because you can argue that permission would be granted anyway.
- pdimitar 8y agoWell I am not arguing that point because I am not doing unethical scraping anyway. Just trying to explain why most scrapers go to hammering the servers directly. Additionally, in my local market the owners of e-commerce websites are extremely narrow-minded and have zero tech education so all they will ever hear from you is "I want to steal that guy's data" which is of course not true at all. But try and argue with a 50-year old guy with the mindset of a feudal master who never truly worked in their life but want to control how everybody around them works. If the survival of my business was at stake, I would just scrape one page every 3 or so seconds as a reasonable compromise. In fact I have done so for my amateur scraping experiments, although there the timeout was even steeper -- 10 seconds per page.
- ryandrake 8y agoI did a project in the past that involved scraping non-mainstream e-commerce sites and we encountered this mindset. Durr, what? Yer want to take all mah data?? It ended up easier to just write the scrapers than to explain what we were doing to Neanderthals.
- Theodores 8y agoI forgot about the managers! They get to have jobs this way.
- pdimitar 8y ago> Durr, what? Yer want to take all mah data?? It ended up easier to just write the scrapers than to explain what we were doing to Neanderthals. As demeaning and offensive many people would find that statement to be, I still found it to be the sad reality most of the time. Plus my local community is much smaller and I would not want vengeful businessmen who understand NOTHING from what I am trying to achieve, to actively sabotage me. They can easily call my ISP and deny me service, for example. So I opted for ethical scraping without asking questions. Seems to be the best working compromise. Thanks for sharing your experience. Let's bathe in the confirmation bias it dips us in. :D