5 ms·
Is it common for developers in the eCommerce space to use scrapers as a means to aggressively push automated price-match algorithms? I've been asked to do this
by fareesh 10y ago
Is it common for developers in the eCommerce space to use scrapers as a means to aggressively push automated price-match algorithms? I've been asked to do this a number of times, was just curious as to how prevalent it is.
- hk__2 10y agoYes, everybody scrapes the prices of the others.
- rckclmbr 10y agoMost companies will use resources not in their datacenter and not identifiable. Executives know it's sketchy, but they do it anyway.
- fareesh 10y agoWith stuff like Facebook opengraph (e.g. og:price) and other meta tags meant to help search engines and social networks get this sort of data to display inline, do you think it's inevitable that complex scraping will no longer be needed in a practical sense since everyone will be inadvertently optimizing their markup in a way that you could write a really simple parser to grab the data?
- greglindahl 10y agoGiven that FB's bot identifies itself, no, eventually some websites will present og: markup only to FB's bot.
- fareesh 10y agoCouldn't you just spoof the user agent though? Or is there some other mechanism that you can use to verify the bot is Facebook's?
- greglindahl 10y agoIP address.