3 ms·

> I've become so reliant on CCC it really bugs me I can't use it (or an equivalent) on other sites -- e.g. Target, Wal-Mart, Best Buy, Nike, J. Crew, whatever.

> I've become so reliant on CCC it really bugs me I can't use it (or an equivalent) on other sites -- e.g. Target, Wal-Mart, Best Buy, Nike, J. Crew, whatever. I've actually been toying with this concept for a weekend project, essentially a quick 'n dirty node daemon process that checks prices periodically, for sites it knows how to. More often than not the data is just sitting there in JSON in the <head>, so you wouldn't necessarily have to scrape it either. (e.g. fossil) Open source the project to let others send PRs for other sites... Toss in a sendgrid key for emails and you're all set...

bransonf 6y ago
I think the only flaw here is that while it may be trivial to get raw data from say fossil, any major multi-department retailer (Amazon, Walmart, Best Buy, Target, B&H...) is not going to be this easy. Prices are competitive information, at least competitive enough to warrant obfuscation and preventing scraping. And, the more sources you have, the more likely things will constantly fail. Scraping is always fragile. One of the first web-scraping projects I ever built was to find the lowest price for Clif bars. I ate about 365 per year, and in only 45 minutes of programming a cron job and push notifications, I easily saved $100 that year.
- zikzak 6y ago
  As someone who may or may not monitor very elastic competitor pricing (allegedly) I would add that if you are not respectful with your scraping you will get IP banned. You might also get shut down by a provider (e.g. aws). Services like Crawlera act as a proxy that also forces you to be nice. If anyone is getting into this, I would recommend they use a service like that to scrape at scale.
immewnity 6y ago
Sounds similar to what Klaxon does. https://github.com/themarshallproject/klaxon https://github.com/themarshallproject/klaxon
crazygringo 6y ago
Honestly I'd prefer if the process didn't even have to "know" sites. Let me just supply the URL and the price string... and let it automatically figure out the right <div> or class or id or surrounding text it can be extracted from. And if it ever breaks (can't be found, or returns something in a different numeric format) then just notify me to visit it and see what's up. Command-line is one way to do it (can be the canonical version), but then a browser extension makes it even easier. You'll need a headless browser even from the command-line in any case, since the product info isn't always in the HTML at all.

bransonf 6y ago

I think the only flaw here is that while it may be trivial to get raw data from say fossil, any major multi-department retailer (Amazon, Walmart, Best Buy, Target, B&H...) is not going to be this easy. Prices are competitive information, at least competitive enough to warrant obfuscation and preventing scraping. And, the more sources you have, the more likely things will constantly fail. Scraping is always fragile. One of the first web-scraping projects I ever built was to find the lowest price for Clif bars. I ate about 365 per year, and in only 45 minutes of programming a cron job and push notifications, I easily saved $100 that year.

zikzak 6y ago
As someone who may or may not monitor very elastic competitor pricing (allegedly) I would add that if you are not respectful with your scraping you will get IP banned. You might also get shut down by a provider (e.g. aws). Services like Crawlera act as a proxy that also forces you to be nice. If anyone is getting into this, I would recommend they use a service like that to scrape at scale.

immewnity 6y ago

Sounds similar to what Klaxon does. https://github.com/themarshallproject/klaxon https://github.com/themarshallproject/klaxon

crazygringo 6y ago

Honestly I'd prefer if the process didn't even have to "know" sites. Let me just supply the URL and the price string... and let it automatically figure out the right <div> or class or id or surrounding text it can be extracted from. And if it ever breaks (can't be found, or returns something in a different numeric format) then just notify me to visit it and see what's up. Command-line is one way to do it (can be the canonical version), but then a browser extension makes it even easier. You'll need a headless browser even from the command-line in any case, since the product info isn't always in the HTML at all.