4 ms·
Have you considered setting up an API to allow the bots to get what they want without hammering your front-end servers?
by z3c0 5y ago
Have you considered setting up an API to allow the bots to get what they want without hammering your front-end servers?
- kayodelycaon 5y agoWhat makes you think they would use it?
- z3c0 5y agoIt's in their interest. I've scraped a lot, and it's not easy to build a reliable process on. Why parse a human interface when there's an application interface available?
- CWuestefeld 5y agoYes. And if I could get the perpetrators to raise their hands so I could work out an API for them, it would be the path of least resistance. But they take great pains to be anonymous, although I know from circumstantial evidence that at least a good chunk of it is various competitors (or services acting on behalf of competitors) scraping price data. IANAL, but I also wonder if, given that I'd be designing something specifically for competitors to query our prices in order to adjust their own prices, this would constitute some form of illegal collusion.
- marginalia_nu 5y agoWhat seems to actually work is to identify the bots and instead of giving up your hand by blocking them, to quietly poison the data. Critically, it needs to be subtle enough that it's not immediately obvious the data is manipulated. It should look like a plausible response, only with some random changes.