3 ms·

All very valid reasons (except to a degree the "screwing with metrics" one). There a lot of websites which do not really face any of the aforementioned issues,

All very valid reasons (except to a degree the "screwing with metrics" one). There a lot of websites which do not really face any of the aforementioned issues, simply because they do not sell you anything, are not an ad network or run referral programs, do not even have user-generated content, etc. And even the sites that do, it's usually a rather small part of the "surface" that needs such protections e.g. the actual API call to make a checkout or post a comment. And yet if you dare browse the web with TOR or a VPN or sometimes just happen to be on a small ISP[0][1] then you're being punished immediately. You solve your cloudflare-supplied captcha because you may be a bot (you're not, and the dangerous bots will not be defeated by this anyway, but some humans will be), and then you get an error from the website itself because it runs a secondary bot detection thing. And you weren't even anywhere near anything "dangerous" like a checkout page. [0] My parents use a regional small ISP (but locally very popular) that serves around 50K customers. My parents also use a regional bank (a Volksbank, and those are members of a national association that provides all kinds of services). Suddenly that bank would not even let them see the bank's front page. After some back and forth on the phone support line it turned out the bank had recently deployed some "advanced" bot detection, one that had a whitelist of residential-ISP-associated AS/IP ranges, and of course whoever compiled and maintained that list had forgot to include that small local ISP. For that regional bank it meant they had just shut out a very significant number of their customers (and potential customers just trying to look up what the bank offers), as there was very likely a huge overlap of people using that regional ISP and that regional bank (both are regional, after all). It also was something they couldn't fix themselves, as the "online banking" stuff was not in-house but was run by the national association (which probably used some bot-detection as a service provider). It took the bank (or rather, the national association) a few weeks to fix. Mind you, the last few years that bank has been heavily marketing a cheaper "online only" account, only online banking, online support, and access to the self-serve ATMs and banking terminals, but no face-to-face or even ear-to-ear human interaction. Try contacting "online support" about "the website outright refuses me" when the "online support" is only available on that website. Kudos is you're smart enough to switch to their mobile app, as your phone uses a different ISP, unless you forget to turn off wifi. That's the advice my parents got from the phone support (they sill have an account type that not online-only). [1] When I recently visited my parents, from the wifi [same ISP as in 0] I couldn't open the website of a bakery too look up if and when they would be open on a Sunday. Some error message about "this website is not available in your network" (English text, for a German bakery... suspicious :P). I could open it via my mobile, tho. I could open it from my regular ISP when back home (in another city) again. Mind you, that website is purely informational and has no "interactive" features let alone let's you buy anything. It's just static text and some pictures.

maven29 5y ago
It doesn't help that VPN providers in the consumer space pay small ISPs to front their traffic so that netflix et al doesn't drop their traffic coming from a datacenter and they still get to keep up the false pretenses and get valued at billions of dollars. Smaller ISPs are also more likely to have issues with CPE getting compromised and routers running botnets within the comfort of your home. There are services which invite people to sell their residential bandwidth in return for money, this can potentially have a disproportionate impact at smaller sample sizes. Networks can declare themselves to be ISPs. You could check if your ISP shows up as an ISP in peeringDB. https://networkengineering.stackexchange.com/questions/44585/how-to-tell-if-an-asn-belongs-to-a-hosting-company-isp-or-to-a-regular-organiz https://networkengineering.stackexchange.com/questions/44585...
- rndgermandude 5y ago
  Yeah, but that's not what happens with this small German ISP my parents use, which was started because some people and local businesses really got disgruntled at the pricing policy and lack of true broadband by the Deutsche Telekom. This thing is essentially run like a non-profit, the little they make in profit meant to be invested into the company again not to make some investors happy, and is majority owned by the city-owned municipal utilities company, with the rest of the ownership I believe in the hands of some local businesses (some of which quite large) who needed broadband but couldn't get it or only at astronomical prices; they are their own customers and thus not very much inclined to fuck themselves. They are reportedly very proactive when it comes to CPE security, as well, up to giving customers a proactive phone call when they see somebody is using equipment with known vulnerabilities (customers are allowed to operate their own equipment as long as it is deemed compatible, most will use remotely managed equipment, tho, I believe; my dad used to use his own DSL router and once got such a call if I remember correctly. He switched over to their fiber now and managed equipment). Their AS is indeed identified as an "ISP" in peeringDB. While I would be extremely surprised if the company was doing shady things, you surely got a point that a small ISP like that could suffer more in reputation from some few customers being up to shady things, including sub-lending the line. I am pretty sure that is against the ToS, but enforcement is a problem of course. Especially detecting such traffic without violating German privacy laws is probably a difficult task, but it's not impossible.

maven29 5y ago

It doesn't help that VPN providers in the consumer space pay small ISPs to front their traffic so that netflix et al doesn't drop their traffic coming from a datacenter and they still get to keep up the false pretenses and get valued at billions of dollars. Smaller ISPs are also more likely to have issues with CPE getting compromised and routers running botnets within the comfort of your home. There are services which invite people to sell their residential bandwidth in return for money, this can potentially have a disproportionate impact at smaller sample sizes. Networks can declare themselves to be ISPs. You could check if your ISP shows up as an ISP in peeringDB. https://networkengineering.stackexchange.com/questions/44585/how-to-tell-if-an-asn-belongs-to-a-hosting-company-isp-or-to-a-regular-organiz https://networkengineering.stackexchange.com/questions/44585...

rndgermandude 5y ago
Yeah, but that's not what happens with this small German ISP my parents use, which was started because some people and local businesses really got disgruntled at the pricing policy and lack of true broadband by the Deutsche Telekom. This thing is essentially run like a non-profit, the little they make in profit meant to be invested into the company again not to make some investors happy, and is majority owned by the city-owned municipal utilities company, with the rest of the ownership I believe in the hands of some local businesses (some of which quite large) who needed broadband but couldn't get it or only at astronomical prices; they are their own customers and thus not very much inclined to fuck themselves. They are reportedly very proactive when it comes to CPE security, as well, up to giving customers a proactive phone call when they see somebody is using equipment with known vulnerabilities (customers are allowed to operate their own equipment as long as it is deemed compatible, most will use remotely managed equipment, tho, I believe; my dad used to use his own DSL router and once got such a call if I remember correctly. He switched over to their fiber now and managed equipment). Their AS is indeed identified as an "ISP" in peeringDB. While I would be extremely surprised if the company was doing shady things, you surely got a point that a small ISP like that could suffer more in reputation from some few customers being up to shady things, including sub-lending the line. I am pretty sure that is against the ToS, but enforcement is a problem of course. Especially detecting such traffic without violating German privacy laws is probably a difficult task, but it's not impossible.