3 ms·
What's the consequence of installing the crawler to FF? Can the ISP / Cloudflare / any other party start blacklisting you?
by Alifatisk 3y ago
What's the consequence of installing the crawler to FF? Can the ISP / Cloudflare / any other party start blacklisting you?
- KomoD 3y ago> Can the ISP / Cloudflare / any other party start blacklisting you? Yes, wouldn't surprise me if you were to get blocked from tons of sites and flagged as a bot on Cloudflare and other WAFs.
- Alifatisk 3y agoYeah getting blocked from Cloudflares network is not something I want to experience, it's a hell.
- hinkley 3y agoWell, the internet could know the balance of your bank account at that boutique credit union, or which sex toys you’ve been looking at buying. We do web hosting for businesses. We are careful to identify bots because we don’t want personalized recommendations in search engines, and don’t want PII showing up. Not that we put a lot of that in the site, but think about a Home Depot or a Best Buy situation. If you’re a human near the Minneapolis store, you want availability for that store, not the Detroit one. But corporate wants a bot to know that this microwave is something you in theory carry, but might have to ship from a warehouse. If Home Depot wants that, and our customers also want that, then probably a lot of businesses feel the same.
- Alifatisk 3y agoBut the extension is not crawling my stuff, it takes batches of urls from the host server. How is the internet going to get my bank account?