4 ms·

Show HN: BrowseNodes – Get unthrottled access to Amazon Product API

8 points by BrowseNodes 8y ago

BrowseNodes 8y ago
When I was building a price comparison site using Amazon Product Advertising API, I ran into this throttling error frequently: "You are submitting requests too quickly. Please retry your requests at a slower rate." So, I built a shared cache, so that developers can hit the cache instead of hitting Amazon API every time. That's how I created BrowseNodes. Have you faced the same problem like me? Do you think this will solve the throttling problem for you? Looking forward to your feedback!
visionerdy 8y ago
According to the Amazon API docs, we shouldn't be sharing the API key. Do you store our API key in the cache?
- BrowseNodes 8y ago
  No. I don't store your API key (i.e. awsSecret). I only cache the product data to serve other users: https://browsenodes.com/how-to-solve-requestthrottled.html https://browsenodes.com/how-to-solve-requestthrottled.html Hope that helps!
gitgud 8y ago
If this just caches the requests, then couldn't you still be throttled? This just moves the throttling problem to BrowserNode
- BrowseNodes 8y ago
  Not really! Lets say, user 1 makes an API call to our server to fetch browse node: 1000 (Books). The first time such request is made, our server will make an API call to Amazon using user 1's AWS key. Once we get the response back, we will cache the result and send back the result to user 1. When user 2 makes the request for the same browse node 1000 (Books), we will not hit Amazon server again. Instead, we will get the cached data from our database and send it back to the user 2 without using his/her AWS key. This way the number of API calls made will be less than one per second for a user.
  - chatmasta 8y ago
    This assumes all your users are searching for the same browse node. I’m not familiar with the Amazon API, how many browse nodes are there?
    - BrowseNodes 8y ago
      Yes. Browse Nodes are essentially product categories. So, users search for the same set of categories. There are ~40K browse nodes in each Amazon domain and we have cached most of them already [1]. So, the cache miss is unlikely here. But, product cache is little complex. We have already cached the top products [2]. But, products are added every day. So, there will be some amount of cache miss there. [1] Cached browse nodes - https://browsenodes.com/amazon.com/explore.html https://browsenodes.com/amazon.com/explore.html [2] Cached products - https://browsenodes.com/amazon.co.uk/itemSearch/83451031.html?parent=Apparel https://browsenodes.com/amazon.co.uk/itemSearch/83451031.htm...
      - gitgud 8y ago
        Ah I see now, so you guys have pre-fetched most of Amazon and are providing a less restricted API to the cached content. Is that permitted in the Terms and Service of the Amazon API? Anyway great idea, would make a lot of developers lives easier!
        BrowseNodes 8y ago
        Thanks! Amazon TOS allows caching. But, not sure if there is any specific restriction on shared cache.
  - deleted 8y ago
    [deleted]

8 points by BrowseNodes 8y ago

When I was building a price comparison site using Amazon Product Advertising API, I ran into this throttling error frequently: "You are submitting requests too quickly. Please retry your requests at a slower rate." So, I built a shared cache, so that developers can hit the cache instead of hitting Amazon API every time. That's how I created BrowseNodes. Have you faced the same problem like me? Do you think this will solve the throttling problem for you? Looking forward to your feedback!

visionerdy 8y ago

According to the Amazon API docs, we shouldn't be sharing the API key. Do you store our API key in the cache?

BrowseNodes 8y ago
No. I don't store your API key (i.e. awsSecret). I only cache the product data to serve other users: https://browsenodes.com/how-to-solve-requestthrottled.html https://browsenodes.com/how-to-solve-requestthrottled.html Hope that helps!

gitgud 8y ago

If this just caches the requests, then couldn't you still be throttled? This just moves the throttling problem to BrowserNode

BrowseNodes 8y ago
Not really! Lets say, user 1 makes an API call to our server to fetch browse node: 1000 (Books). The first time such request is made, our server will make an API call to Amazon using user 1's AWS key. Once we get the response back, we will cache the result and send back the result to user 1. When user 2 makes the request for the same browse node 1000 (Books), we will not hit Amazon server again. Instead, we will get the cached data from our database and send it back to the user 2 without using his/her AWS key. This way the number of API calls made will be less than one per second for a user.
- chatmasta 8y ago
  This assumes all your users are searching for the same browse node. I’m not familiar with the Amazon API, how many browse nodes are there?
  - BrowseNodes 8y ago
    Yes. Browse Nodes are essentially product categories. So, users search for the same set of categories. There are ~40K browse nodes in each Amazon domain and we have cached most of them already [1]. So, the cache miss is unlikely here. But, product cache is little complex. We have already cached the top products [2]. But, products are added every day. So, there will be some amount of cache miss there. [1] Cached browse nodes - https://browsenodes.com/amazon.com/explore.html https://browsenodes.com/amazon.com/explore.html [2] Cached products - https://browsenodes.com/amazon.co.uk/itemSearch/83451031.html?parent=Apparel https://browsenodes.com/amazon.co.uk/itemSearch/83451031.htm...
    - gitgud 8y ago
      Ah I see now, so you guys have pre-fetched most of Amazon and are providing a less restricted API to the cached content. Is that permitted in the Terms and Service of the Amazon API? Anyway great idea, would make a lot of developers lives easier!
      - BrowseNodes 8y ago
        Thanks! Amazon TOS allows caching. But, not sure if there is any specific restriction on shared cache.
- deleted 8y ago
  [deleted]