3 ms·
How does this work? Is there a database of the most popular online stores and you're constantly scraping (or GETing) product data?
by jliptzin 14y ago
How does this work? Is there a database of the most popular online stores and you're constantly scraping (or GETing) product data?
- netvarun 14y agoWe aggregate data from a variety of sources (crawling, data dumps, rss feeds, and in some cases even manual curation) after which we integrate them into our data pipeline. We update them using a power law distribution, where the top 1% of best selling products (based on our internal ranking system) is updated hourly, the next 3% updated every two hours, etc.. The whole index is refreshed at the end of each month.
- jliptzin 14y agoVery cool. Thanks for the explanation.