4 ms·
I think the only flaw here is that while it may be trivial to get raw data from say fossil, any major multi-department retailer (Amazon, Walmart, Best Buy, Targ
by bransonf 6y ago
I think the only flaw here is that while it may be trivial to get raw data from say fossil, any major multi-department retailer (Amazon, Walmart, Best Buy, Target, B&H...) is not going to be this easy.
Prices are competitive information, at least competitive enough to warrant obfuscation and preventing scraping.
And, the more sources you have, the more likely things will constantly fail. Scraping is always fragile.
One of the first web-scraping projects I ever built was to find the lowest price for Clif bars. I ate about 365 per year, and in only 45 minutes of programming a cron job and push notifications, I easily saved $100 that year.
- zikzak 6y agoAs someone who may or may not monitor very elastic competitor pricing (allegedly) I would add that if you are not respectful with your scraping you will get IP banned. You might also get shut down by a provider (e.g. aws). Services like Crawlera act as a proxy that also forces you to be nice. If anyone is getting into this, I would recommend they use a service like that to scrape at scale.