4 ms·

It’s more like service slaughter (which isn’t a term AFAIK) than denial of service (since denial isn’t the goal). It’s a tough problem though. In an ideal worl

It’s more like service slaughter (which isn’t a term AFAIK) than denial of service (since denial isn’t the goal). It’s a tough problem though. In an ideal world, I think scrapers should pay for the compute/resources required to serve their requests. I think that’s one of the only ways to scale for this fairly. (This assumes you agree that scraping of public content should be allowed in the first place.)

jka 6y ago
That's an interesting idea. Sometimes a lot of the (service provision) cost depends on how efficient the data access is at the host. For example, let's take an API that includes a 'GET /data?start=0&end=100' endpoint that retrieves items with key between 0 and 100. If items are stored in sorted key order, that could be an efficient and inexpensive query that retrieves exactly the 100 relevant items and doesn't perform any other unnecessary work. If, on the other hand, the database contains a billion items and they aren't stored in key order, then with a caller-pays pricing model, it's (probably, for the sake of argument) not worth running that query. In some cases it might be reasonable to continue to pay for the expensive queries. But what if all you need to do is convince the host to 'CREATE INDEX ... ON (key)'? (all very hypothetical - I don't have any explicit answers. if the host service were open source, the answer might be 'open a pull request and talk to them, and hopefully reduce everyone's costs')
- ethanwillis 6y ago
  If I was evil and the law was that scrapers had access BUT they had to pay their fair share of the cost. I would just make their costs so absurdly high that it wouldn't be beneficial to them.
  - andyfleming 6y ago
    Yeah, you’d almost need some way to genetically price this. You’d almost want websites to opt into it. You can either accept the $x/request price or not participate in the program. Though, I guess at that point you might as well just sell API access.
- andyfleming 6y ago
  Yeah, a hard part of this is that it depends on the efficiency of the implementation by the website creators/maintainers. At least at that point you’re on the same side. You’re both trying to be fair and you’re both trying to help the website work well.

jka 6y ago

That's an interesting idea. Sometimes a lot of the (service provision) cost depends on how efficient the data access is at the host. For example, let's take an API that includes a 'GET /data?start=0&end=100' endpoint that retrieves items with key between 0 and 100. If items are stored in sorted key order, that could be an efficient and inexpensive query that retrieves exactly the 100 relevant items and doesn't perform any other unnecessary work. If, on the other hand, the database contains a billion items and they aren't stored in key order, then with a caller-pays pricing model, it's (probably, for the sake of argument) not worth running that query. In some cases it might be reasonable to continue to pay for the expensive queries. But what if all you need to do is convince the host to 'CREATE INDEX ... ON (key)'? (all very hypothetical - I don't have any explicit answers. if the host service were open source, the answer might be 'open a pull request and talk to them, and hopefully reduce everyone's costs')

ethanwillis 6y ago
If I was evil and the law was that scrapers had access BUT they had to pay their fair share of the cost. I would just make their costs so absurdly high that it wouldn't be beneficial to them.
- andyfleming 6y ago
  Yeah, you’d almost need some way to genetically price this. You’d almost want websites to opt into it. You can either accept the $x/request price or not participate in the program. Though, I guess at that point you might as well just sell API access.
andyfleming 6y ago
Yeah, a hard part of this is that it depends on the efficiency of the implementation by the website creators/maintainers. At least at that point you’re on the same side. You’re both trying to be fair and you’re both trying to help the website work well.