4 ms·

No. When I say that bots exceed the amount of real traffic, I'm including people "window shopping" on the good side. My complaint is more like, somebody wants

No. When I say that bots exceed the amount of real traffic, I'm including people "window shopping" on the good side. My complaint is more like, somebody wants to know the prices of all our products, and that we have roughly X products (where X is a very large number). They get X friends to all go into the store almost simultaneously, each writing down the price of the particular product they've been assigned to research. When they do this, there's scant space left in the store for even the browsing kind of customers to walk in. (of course I exaggerate a bit, but that's the idea)

mrobins 5y ago
I’m sympathetic to the complaints about “rude” scraping behavior but there’s an easy solution. Rather than make people consume boatloads of resources they don’t want (individual page views, images, scripts, etc.) just build good interoperability tools that give the people what they want. In the physical example above that would be a product catalog that’s easily replicated with a CSV product listing or an API.
- jensensbutton 5y ago
  You don't know why any random scraper is scraping you and thus you don't know what api to build that will do them from scraping. Also, it's likely easier for them to contribute scraping than write a bunch of code to integrate with your API so it there's no incentive for them to do so either.
  - altdataseller 5y ago
    For the 2nd part, I have done scraping and would always opt for an API if the price is reasonable over paying nosebleed amounts for residential proxies
  - CWuestefeld 5y ago
    Yes, exactly. Nobody is standing up and saying "we're the ones doing this, and here's what we wish you'd put in an API". Also, I'm a big Jenson Button fan.
  - mrobins 5y ago
    I think lots of website owners know exactly where the value in their content exists. Whether or not they want to share that in a convenient way, especially to competitors etc is another story. That said if scraping is inevitable, it’s immensely wasteful effort to both the scraper and the content owner that’s often avoidable.
  - mcronce 5y ago
    Writing a scraper for a webpage is typically far more development effort than writing an API wrapper
    - johnnyanmac 5y ago
      Yes, but the scraper in this context is already built. A bird in hand and all that.
      - mcronce 5y ago
        In this case, yes, obviously. But as far as "it's likely easier for them to contribute scraping than write a bunch of code to integrate with your API", that presupposes no existing integration.
  - withinboredom 5y ago
    Just advertise the API in the headers. Or better yet, set the buttons/links only to be accessible via .usetheapi-dammit selector. Lastly, provide an API and a “developers.whatever.com” domain to report issues with the API, get API keys, and pay for more requests. It should be pretty easy to setup, especially if there’s an internal API available behind the frontend already. I’d venture a dev team could devote 20% to a few sprints and have an MVP thing up and running.

mrobins 5y ago

I’m sympathetic to the complaints about “rude” scraping behavior but there’s an easy solution. Rather than make people consume boatloads of resources they don’t want (individual page views, images, scripts, etc.) just build good interoperability tools that give the people what they want. In the physical example above that would be a product catalog that’s easily replicated with a CSV product listing or an API.

jensensbutton 5y ago
You don't know why any random scraper is scraping you and thus you don't know what api to build that will do them from scraping. Also, it's likely easier for them to contribute scraping than write a bunch of code to integrate with your API so it there's no incentive for them to do so either.
- altdataseller 5y ago
  For the 2nd part, I have done scraping and would always opt for an API if the price is reasonable over paying nosebleed amounts for residential proxies
- CWuestefeld 5y ago
  Yes, exactly. Nobody is standing up and saying "we're the ones doing this, and here's what we wish you'd put in an API". Also, I'm a big Jenson Button fan.
- mrobins 5y ago
  I think lots of website owners know exactly where the value in their content exists. Whether or not they want to share that in a convenient way, especially to competitors etc is another story. That said if scraping is inevitable, it’s immensely wasteful effort to both the scraper and the content owner that’s often avoidable.
- mcronce 5y ago
  Writing a scraper for a webpage is typically far more development effort than writing an API wrapper
  - johnnyanmac 5y ago
    Yes, but the scraper in this context is already built. A bird in hand and all that.
    - mcronce 5y ago
      In this case, yes, obviously. But as far as "it's likely easier for them to contribute scraping than write a bunch of code to integrate with your API", that presupposes no existing integration.
- withinboredom 5y ago
  Just advertise the API in the headers. Or better yet, set the buttons/links only to be accessible via .usetheapi-dammit selector. Lastly, provide an API and a “developers.whatever.com” domain to report issues with the API, get API keys, and pay for more requests. It should be pretty easy to setup, especially if there’s an internal API available behind the frontend already. I’d venture a dev team could devote 20% to a few sprints and have an MVP thing up and running.