3 ms·
I've worked on elaborate projects with the goal of world domination over the past few years, and all have failed. Now, I'm working on building a portfolio of sm
by brildum 15y ago
I've worked on elaborate projects with the goal of world domination over the past few years, and all have failed. Now, I'm working on building a portfolio of smaller apps to slowly build passive income (with the potential for a big win).
TidyFinder is the first step towards that goal. I'd appreciate any feedback/criticism/suggestions that you can offer. I value the HN community's thoughts. Thanks.
- philfreo 15y agoWould love to hear some details on how you built it / got your data
- brildum 15y agoSome basic details below, maybe I'll write a blog post with more thorough descriptions... App and scrapers are written in Python with MongoDB storage. I was pleasantly surprised with MongoDB's query speed given I only index keywords. There are a small number of products in the database, but its still very fast. To get the data I've created scrapers for each site. Currently, only Target and Amazon are implemented (with Container Store soon to come). Target and Container Store provide a product feed when you sign up as an affiliate. They FTP their product info to your site and I use that to parse the links. Using each link, I then scrape the page to store the relevant information for various products in the database. Amazon provides a nice API and there is a Python wrapper which makes it easy to query (python-amazon-product-api).
- Luyt 15y agoI guess you scrape websites of retailers to populate your database? Have there been any retailers which said you are not allowed to scrape their site for gathering product info, pictures and prices? (I can imagine that certain retailers only want their products to be showcased in the context of their own site)