4 ms·

Doesn't Amazon have an API? Why everyone wants to scrape Amazon? Honest question.

helsinkiandrew 5y ago
Yes, all their product data is available for affiliates: https://webservices.amazon.com/paapi5/documentation/ https://webservices.amazon.com/paapi5/documentation/
- temp8964 5y ago
  Also, the document says "Product Advertising API is free". I am confused.
  - moltar 5y ago
    The access is only provided to active affiliates with sales. Amount of access is counted by number of sales too. More sales = more API access.
    - temp8964 5y ago
      Thanks. This is very helpful. This information is not provided on the documentation.
      - tethys 5y ago
        API rates can be found here: https://webservices.amazon.com/paapi5/documentation/troubleshooting/api-rates.html https://webservices.amazon.com/paapi5/documentation/troubles...
        temp8964 5y ago
        Thanks. This is really annoying. Now I understand why people just want to scrape.
mkl95 5y ago
Amazon's marketplace APIs are available to developers who register through a seller account. Pricing is $40 per month. The main benefit of using the API is that you can request a LOT of data without hitting their rate limit. Unless you need to get dozens of results per second, you are usually better off with a spider (or just use Huginn). And if you are hit with a 503 and a captcha, gluing a free captcha solver with some middleware is a trivial task.
- temp8964 5y ago
  But other comments say scraping Amazon is kind of complicated because they ban IPs? I am not sure if you have a seller / affiliate account and then use your home IP to do scaping, will that impact your seller / affiliate account ?
  - mkl95 5y ago
    You shouldn't use the same IP to continuously scrape Amazon. Personally I use a $8/month rotating proxy service that gives me a new proxy list every hour (webshare.io if it piques your interest, I'm in no way associated with them). Also, in my experience Amazon's "ban" comes down to solving a captcha on every request, so it's more like some mild throttling than a real ban.
- moehm 5y ago
  > free captcha solver I haven't looked into this space, but what is a free captcha solver? And what is the drawback? Wouldn't this defeat the purpose of a captcha?
  - DoctorOW 5y ago
    I also haven't looked into this space outside of a quick DDG search after reading your comment. It looks like the big hole is that Amazon rolled their own captcha a while ago and haven't kept up with what automation can do now.
  - mkl95 5y ago
    I'm using a ML-based captcha solver that is free software available on Github. So far it has solved 100% of the Amazon captchas it has encountered. The reason Amazon have a reputation of being good at blocking IPs is that their responses are (purposefully?) obscure. The way it works, it filters out script kiddies and lets engineers through, which probably are a small minority of the people scraping Amazon.
avipars 5y ago
Amazon has one for their affiliates, you need to be approved and get 3 sales attributed to you in order to apply.

Yes, all their product data is available for affiliates: https://webservices.amazon.com/paapi5/documentation/ https://webservices.amazon.com/paapi5/documentation/

temp8964 5y ago
Also, the document says "Product Advertising API is free". I am confused.
- moltar 5y ago
  The access is only provided to active affiliates with sales. Amount of access is counted by number of sales too. More sales = more API access.
  - temp8964 5y ago
    Thanks. This is very helpful. This information is not provided on the documentation.
    - tethys 5y ago
      API rates can be found here: https://webservices.amazon.com/paapi5/documentation/troubleshooting/api-rates.html https://webservices.amazon.com/paapi5/documentation/troubles...
      - temp8964 5y ago
        Thanks. This is really annoying. Now I understand why people just want to scrape.

mkl95 5y ago

Amazon's marketplace APIs are available to developers who register through a seller account. Pricing is $40 per month. The main benefit of using the API is that you can request a LOT of data without hitting their rate limit. Unless you need to get dozens of results per second, you are usually better off with a spider (or just use Huginn). And if you are hit with a 503 and a captcha, gluing a free captcha solver with some middleware is a trivial task.

temp8964 5y ago
But other comments say scraping Amazon is kind of complicated because they ban IPs? I am not sure if you have a seller / affiliate account and then use your home IP to do scaping, will that impact your seller / affiliate account ?
- mkl95 5y ago
  You shouldn't use the same IP to continuously scrape Amazon. Personally I use a $8/month rotating proxy service that gives me a new proxy list every hour (webshare.io if it piques your interest, I'm in no way associated with them). Also, in my experience Amazon's "ban" comes down to solving a captcha on every request, so it's more like some mild throttling than a real ban.
moehm 5y ago
> free captcha solver I haven't looked into this space, but what is a free captcha solver? And what is the drawback? Wouldn't this defeat the purpose of a captcha?
- DoctorOW 5y ago
  I also haven't looked into this space outside of a quick DDG search after reading your comment. It looks like the big hole is that Amazon rolled their own captcha a while ago and haven't kept up with what automation can do now.
- mkl95 5y ago
  I'm using a ML-based captcha solver that is free software available on Github. So far it has solved 100% of the Amazon captchas it has encountered. The reason Amazon have a reputation of being good at blocking IPs is that their responses are (purposefully?) obscure. The way it works, it filters out script kiddies and lets engineers through, which probably are a small minority of the people scraping Amazon.

avipars 5y ago

Amazon has one for their affiliates, you need to be approved and get 3 sales attributed to you in order to apply.