4 ms·

I have sympathy for your operational issues and costs, but isn't this kind of complaint the same as a shopping mall/center complaining of people who go in, chec

I have sympathy for your operational issues and costs, but isn't this kind of complaint the same as a shopping mall/center complaining of people who go in, check some info and go out without buying? I understand that bots have leverage and automation, but so does you to reach a larger audience. Should we continue to benefit from one side of the leverage, while complaining about the other side?

wlesieutre 5y ago
It's more like a mall complaining that while they're trying to serve 1000 customers, someone has gone and dumped 10000000 roombas throughout the stores which are going around scanning all the price tags.
CWuestefeld 5y ago
No. When I say that bots exceed the amount of real traffic, I'm including people "window shopping" on the good side. My complaint is more like, somebody wants to know the prices of all our products, and that we have roughly X products (where X is a very large number). They get X friends to all go into the store almost simultaneously, each writing down the price of the particular product they've been assigned to research. When they do this, there's scant space left in the store for even the browsing kind of customers to walk in. (of course I exaggerate a bit, but that's the idea)
- mrobins 5y ago
  I’m sympathetic to the complaints about “rude” scraping behavior but there’s an easy solution. Rather than make people consume boatloads of resources they don’t want (individual page views, images, scripts, etc.) just build good interoperability tools that give the people what they want. In the physical example above that would be a product catalog that’s easily replicated with a CSV product listing or an API.
  - jensensbutton 5y ago
    You don't know why any random scraper is scraping you and thus you don't know what api to build that will do them from scraping. Also, it's likely easier for them to contribute scraping than write a bunch of code to integrate with your API so it there's no incentive for them to do so either.
    - altdataseller 5y ago
      For the 2nd part, I have done scraping and would always opt for an API if the price is reasonable over paying nosebleed amounts for residential proxies
    - CWuestefeld 5y ago
      Yes, exactly. Nobody is standing up and saying "we're the ones doing this, and here's what we wish you'd put in an API". Also, I'm a big Jenson Button fan.
    - mrobins 5y ago
      I think lots of website owners know exactly where the value in their content exists. Whether or not they want to share that in a convenient way, especially to competitors etc is another story. That said if scraping is inevitable, it’s immensely wasteful effort to both the scraper and the content owner that’s often avoidable.
    - mcronce 5y ago
      Writing a scraper for a webpage is typically far more development effort than writing an API wrapper
      - johnnyanmac 5y ago
        Yes, but the scraper in this context is already built. A bird in hand and all that.
        mcronce 5y ago
        In this case, yes, obviously. But as far as "it's likely easier for them to contribute scraping than write a bunch of code to integrate with your API", that presupposes no existing integration.
kortilla 5y ago
No, because those are people going to the mall. Not robots 100x the quantity of real people.
- signatoremo 5y ago
  Unlike the mall a website can scale up to serve more users at relatively low cost. Also those robots may bring more people to the website. Potentially a lot more even.

wlesieutre 5y ago

It's more like a mall complaining that while they're trying to serve 1000 customers, someone has gone and dumped 10000000 roombas throughout the stores which are going around scanning all the price tags.

CWuestefeld 5y ago

No. When I say that bots exceed the amount of real traffic, I'm including people "window shopping" on the good side. My complaint is more like, somebody wants to know the prices of all our products, and that we have roughly X products (where X is a very large number). They get X friends to all go into the store almost simultaneously, each writing down the price of the particular product they've been assigned to research. When they do this, there's scant space left in the store for even the browsing kind of customers to walk in. (of course I exaggerate a bit, but that's the idea)

mrobins 5y ago
I’m sympathetic to the complaints about “rude” scraping behavior but there’s an easy solution. Rather than make people consume boatloads of resources they don’t want (individual page views, images, scripts, etc.) just build good interoperability tools that give the people what they want. In the physical example above that would be a product catalog that’s easily replicated with a CSV product listing or an API.
- jensensbutton 5y ago
  You don't know why any random scraper is scraping you and thus you don't know what api to build that will do them from scraping. Also, it's likely easier for them to contribute scraping than write a bunch of code to integrate with your API so it there's no incentive for them to do so either.
  - altdataseller 5y ago
    For the 2nd part, I have done scraping and would always opt for an API if the price is reasonable over paying nosebleed amounts for residential proxies
  - CWuestefeld 5y ago
    Yes, exactly. Nobody is standing up and saying "we're the ones doing this, and here's what we wish you'd put in an API". Also, I'm a big Jenson Button fan.
  - mrobins 5y ago
    I think lots of website owners know exactly where the value in their content exists. Whether or not they want to share that in a convenient way, especially to competitors etc is another story. That said if scraping is inevitable, it’s immensely wasteful effort to both the scraper and the content owner that’s often avoidable.
  - mcronce 5y ago
    Writing a scraper for a webpage is typically far more development effort than writing an API wrapper
    - johnnyanmac 5y ago
      Yes, but the scraper in this context is already built. A bird in hand and all that.
      - mcronce 5y ago
        In this case, yes, obviously. But as far as "it's likely easier for them to contribute scraping than write a bunch of code to integrate with your API", that presupposes no existing integration.

kortilla 5y ago

No, because those are people going to the mall. Not robots 100x the quantity of real people.

signatoremo 5y ago
Unlike the mall a website can scale up to serve more users at relatively low cost. Also those robots may bring more people to the website. Potentially a lot more even.