4 ms·

I think I understand, combined with jeffbee's answer, that these sites are behaving selectively according to who you are. So we're back to "No Blacks or Irish"

I think I understand, combined with jeffbee's answer, that these sites are behaving selectively according to who you are. So we're back to "No Blacks or Irish" on the 2022 Internet? What do you think they have against smaller search engines? I can't quite fathom the motives.

wolfgang42 4y ago
There are a lot of crawlers out there, and many of them are ill-behaved. When GoogleBot crawls your site, you get more visitors. When FizzBuzzBot/0.1.3 comes along, you’re more likely to get an overloaded server, weird URLs in your crash logs, spam, or any other manner of mess. Small search engines getting blocked is just collateral damage from websites trying to solve this problem with a blunt ban-hammer.
- bryanrasmussen 4y ago
  This is probably because, as I have learned here on HN, 1 in 4 programmers can't write a decent FizzBuzzBot and should never be hired.
autoexec 4y ago
> So we're back to "No Blacks or Irish" on the 2022 Internet? I think this is the inevitable result of surveillance capitalism. Once you can reliably be identified as holding certain views or meeting some arbitrary criteria pages can be dynamically altered to restrict what you have access to. A racist social platform can look completely benign to anyone who isn't part of the in-group, forums for teens can automatically lock out anyone suspected of being over a certain age, companies can show products or services only to people within a certain class, etc. We already restrict a lot of access to content based on guesses about location. Youtube doesn't even let you see that certain content on their platform exists on their site unless you're logged in from an IP in specific counties (as opposed to showing you the videos exist and throwing up an error saying it's not available for you when you try to view it) and netflix blocks VPNs just to try to keep a bunch of their content away from the wrong kinds of people. These are the kinds of barriers the internet should have freed us from, but instead it's being used to put up more gates and to force people into them.

wolfgang42 4y ago

There are a lot of crawlers out there, and many of them are ill-behaved. When GoogleBot crawls your site, you get more visitors. When FizzBuzzBot/0.1.3 comes along, you’re more likely to get an overloaded server, weird URLs in your crash logs, spam, or any other manner of mess. Small search engines getting blocked is just collateral damage from websites trying to solve this problem with a blunt ban-hammer.

bryanrasmussen 4y ago
This is probably because, as I have learned here on HN, 1 in 4 programmers can't write a decent FizzBuzzBot and should never be hired.

autoexec 4y ago

> So we're back to "No Blacks or Irish" on the 2022 Internet? I think this is the inevitable result of surveillance capitalism. Once you can reliably be identified as holding certain views or meeting some arbitrary criteria pages can be dynamically altered to restrict what you have access to. A racist social platform can look completely benign to anyone who isn't part of the in-group, forums for teens can automatically lock out anyone suspected of being over a certain age, companies can show products or services only to people within a certain class, etc. We already restrict a lot of access to content based on guesses about location. Youtube doesn't even let you see that certain content on their platform exists on their site unless you're logged in from an IP in specific counties (as opposed to showing you the videos exist and throwing up an error saying it's not available for you when you try to view it) and netflix blocks VPNs just to try to keep a bunch of their content away from the wrong kinds of people. These are the kinds of barriers the internet should have freed us from, but instead it's being used to put up more gates and to force people into them.