4 ms·
You don't get to tell site owners what to do. The actual facts on the ground are that they're trying to block your bot. It would be nice if they didn't block yo
by immibis 2y ago
You don't get to tell site owners what to do. The actual facts on the ground are that they're trying to block your bot. It would be nice if they didn't block your bot, but the other, completely unnatural and advertising-driven, monopoly of hosting providers with insane per-request costs makes that impossible until they switch away.
- AlexandrB 2y agoThey try to block your bot because Google is a monopoly and there's little to no cost for blocking everything except Google. This isn't a "natural" monopoly, it's more like Internet Explorer 6.0 and everyone designing their sites to use ActiveX and IE-specific quirks.
- luckylion 2y agoOne possible answer: pay them for their trouble until you provide value to them, e.g. by paying some fraction of a cent for each (document) request.
- BobaFloutist 2y agoCool, you wanna solve micropayments now or wait until we've got cold fusion rolling first...?
- luckylion 2y agoYou wouldn't have to make them micropayments, you can pay out once some threshold is reached. Of course, it would incentivize the sites to make you want to crawl them more, but that might be a good thing. There would be pressure on you to focus on quality over quantity, which would probably be a good thing for your product.
- BobaFloutist 2y ago>You wouldn't have to make them micropayments, you can pay out once some threshold is reached. Believe it or not, this is a potential solution for micropayments that has been explored.
- immibis 2y agoI could even pay a fixed amount to my ISP every month for a fixed amount of data transfer.
- threeseed 2y ago> The actual facts on the ground are that they're trying to block your bot Based on what evidence.
- immibis 2y agobased on them matching the user-agent and sending you a block page? I don't know what else to tell you. It's in plain sight.