2 ms·

It is not just Amazon (although they're #4 on my list of bad actors)[1]. 1. https://files.littlebird.com.au/bad-scrapers.png https://files.littlebird.com.au/ba

It is not just Amazon (although they're #4 on my list of bad actors)[1]. 1. https://files.littlebird.com.au/bad-scrapers.png https://files.littlebird.com.au/bad-scrapers.png

pera 3mo ago
Yeah it's not just Amazon, but so far Amazon was the only one specifically looking for URLs in source code. Interestingly it ignored URLs in the fake markdown docs. Another detail: the scraper did not attempt to access the endpoints immediately (as it did for hrefs in htmls) but it did it on the day after, twice.
- schappim 3mo ago
  Was it actually an Amazon bot IP[1] or someone pretending to be on AWS? 1. https://developer.amazon.com/amazonbot/searchbot-ip-addresses/ https://developer.amazon.com/amazonbot/searchbot-ip-addresse...
  - pera 3mo ago
    Yup, already mentioned in a comment below: the IPs are in that list. They also show up in AbuseIPDB with multiple reports.
cyanydeez 3mo ago
Perhaps you need a TOS that requires a usage fee be mailed to a PO Box that has a reasonable "free" limit of like 100$ and when they exceed it, you automatically mail them a copy of the TOS, the logs and an invoice. Just make sure you have a good lawyer.
- moomoo11 3mo ago
  is there a law they’re breaking? because idk i could be wrong but some small project vs a 2.5T market cap company is gonna need more than “a good lawyer”
  - psd1 3mo ago
    Edit: oops, double-posted.
  - psd1 3mo ago
    It would fall under contact law, and the case would hinge on either the ToS is binding, and whether the bot operator is deemed to have accepted the ToS, and whether access inflicts a cost. In the UK, you would likely win in court, and be awarded £1 for your increased hosting costs. There isn't the concept of punitive damages for contract law here. YMMV. However. People report that you can get paid by sending a plausible invoice to a tech company. The culture disincentivises verifying purchase orders.

It is not just Amazon (although they're #4 on my list of bad actors)[1]. 1. https://files.littlebird.com.au/bad-scrapers.png https://files.littlebird.com.au/bad-scrapers.png

pera 3mo ago

Yeah it's not just Amazon, but so far Amazon was the only one specifically looking for URLs in source code. Interestingly it ignored URLs in the fake markdown docs. Another detail: the scraper did not attempt to access the endpoints immediately (as it did for hrefs in htmls) but it did it on the day after, twice.

schappim 3mo ago
Was it actually an Amazon bot IP[1] or someone pretending to be on AWS? 1. https://developer.amazon.com/amazonbot/searchbot-ip-addresses/ https://developer.amazon.com/amazonbot/searchbot-ip-addresse...
- pera 3mo ago
  Yup, already mentioned in a comment below: the IPs are in that list. They also show up in AbuseIPDB with multiple reports.

cyanydeez 3mo ago

Perhaps you need a TOS that requires a usage fee be mailed to a PO Box that has a reasonable "free" limit of like 100$ and when they exceed it, you automatically mail them a copy of the TOS, the logs and an invoice. Just make sure you have a good lawyer.

moomoo11 3mo ago
is there a law they’re breaking? because idk i could be wrong but some small project vs a 2.5T market cap company is gonna need more than “a good lawyer”
- psd1 3mo ago
  Edit: oops, double-posted.
- psd1 3mo ago
  It would fall under contact law, and the case would hinge on either the ToS is binding, and whether the bot operator is deemed to have accepted the ToS, and whether access inflicts a cost. In the UK, you would likely win in court, and be awarded £1 for your increased hosting costs. There isn't the concept of punitive damages for contract law here. YMMV. However. People report that you can get paid by sending a plausible invoice to a tech company. The culture disincentivises verifying purchase orders.