Just yesterday I wrote a program to scrape some Amazon search and product pages [0]. Why? Because Amazon's search is outright broken. The number of results cha

Just yesterday I wrote a program to scrape some Amazon search and product pages [0]. Why? Because Amazon's search is outright broken. The number of results changes when you change sorting mode, and sometimes, sorting by a different criteria will just serve you a "no products found" error page. I'll generally write a personal product comparison program when it becomes clear that I can't be certain that I can find the best product by hand. Often, even specialized websites that should have parametrized product search/filtering don't have their data properly indexed, so you have to scrape and parse it yourself. Another reason is to cross-reference with data from other sources. E.g. what laptop can I buy that has the best single-threaded CPU performance (within some other restrictions) [1]? [0]: https://github.com/CyberShadow/choose-product/blob/master/amazon/common.d https://github.com/CyberShadow/choose-product/blob/master/am... [1]: https://github.com/CyberShadow/choose-product/blob/master/lenovo/choose.d https://github.com/CyberShadow/choose-product/blob/master/le...