6 ms·
How should they know that your bot belongs to a potential customer and not to a competitor surveilling their prices? Not a rhetorical question, because most co
by carlosjobim 3mo ago
How should they know that your bot belongs to a potential customer and not to a competitor surveilling their prices?
Not a rhetorical question, because most companies would welcome that interest from customers.
- baby_souffle 3mo ago> How should they know that your bot belongs to a potential customer and not to a competitor surveilling their prices? That's the thing; they can't, really. Best we can do is compromise; allow bots and rate limit them. Maybe someday there will be a relatively universal standard for fetching pages akin to the `index.html/index.md` pattern that is emerging now. Put the bot filters on the .html path and some sort of micro transactions system on the .md endpoint. In basically every cat/mouse game defense is forever one step behind...
- Gander5739 3mo agoRate limiting is ineffective against rotating residential proxies, though