Some basic details below, maybe I'll write a blog post with more thorough descriptions... App and scrapers are written in Python with MongoDB storage. I was pl

Some basic details below, maybe I'll write a blog post with more thorough descriptions... App and scrapers are written in Python with MongoDB storage. I was pleasantly surprised with MongoDB's query speed given I only index keywords. There are a small number of products in the database, but its still very fast. To get the data I've created scrapers for each site. Currently, only Target and Amazon are implemented (with Container Store soon to come). Target and Container Store provide a product feed when you sign up as an affiliate. They FTP their product info to your site and I use that to parse the links. Using each link, I then scrape the page to store the relevant information for various products in the database. Amazon provides a nice API and there is a Python wrapper which makes it easy to query (python-amazon-product-api).