6 ms·

I get your point about most people ignoring the absolute shit flood of spurious internet traffic these days. But I don't think anyone can argue that it's the ri

I get your point about most people ignoring the absolute shit flood of spurious internet traffic these days. But I don't think anyone can argue that it's the right thing to do to run a large scale web crawler/indexer with a fake user agent. Google and Bing and others are reasonably good citizens about this. Of course anyone who wants to can set up something that ignores a robots.txt file and leave it as the problem of the httpd operator to either accept or block their traffic. Not caring about false useragents is one thing, caring about somebody running a multi-thousand-requests-per-minute spider requesting pages that no longer exist is another thing. It would still be valid cause for complaint even if it had a valid useragent. The internet would be a better place if more experienced people (such as those who were running httpd 22-24 years ago) reinforced and emphasized the principle, when training and mentoring younger people, that being a good citizen with your network traffic is the responsible and ethical thing to do.

jameshart 4y ago
Nobody’s going to argue that this is an example of a well-behaved bot. I’m just astonished that there are still people who have the time to notice and lament such behavior, and devote time to handcrafting .htaccess rules, rather than just having an aggressive set of heuristics and accepting that some crap will still get through. The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks just feels like a quaint relic of the arpanet.
- robertlagrant 4y ago
  > The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks just feels like a quaint relic of the arpanet. Feelings aside, it doesn't seem that different in principle to an ad blocker with a shared list of ad servers.
- walrus01 4y ago
  > The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks Have you ever worked in network engineering for an ISP that holds a decent sized chunk of ARIN or RIPE ipv4 IP space, and actually cares about its long term IP space reputation? There's a big difference between a $5/mo colo/VPS operator that doesn't give a damn, and an ISP that does pay attention when legit "you have customers with abusive traffic" reports reach their network engineering team. Your comment about not caring about policing badly behaved clients on a network seems like the sort of organization that would land itself on every SMTP RBL in existence. A hosting company (and Azure and AWS are just the giant corporate evolution of a hosting/colo company) needs to be prepared and have a process to fire a customer for violation of TOS/AUP. It's done all the time. The immense size and callous nature of Microsoft/Azure makes this less likely in the modern era.
  - jameshart 4y ago
    SMTP abuse has its mechanisms, with widespread IP blocking mechanisms that ISPs care about. What’s the equivalent mechanism whereby abusive HTTP traffic gets identified and causes an ISP to pay attention? Is there some sort of greybeard grapevine that old school webmasters can use to get the attention of class A IP network operators?
    - walrus01 4y ago
      > Is there some sort of greybeard grapevine that old school webmasters can use to get the attention of class A IP network operators? webmasters as in single operators of httpd somewhere, no, not likely other network operators who are recognized ASNs and go to NANOG or RIPE or APNIC conferences and meet people face to face, yes. if you wilfully disregard valid human-written abuse complaints coming from other equally-clueful network engineering persons at other ISPs, you and your ISP will acquire a tainted reputation. there's various industry back-channels that exist between ISPs to communicate with each other about this sort of thing, I won't go into many details, primarily because there's a high degree of interest of the parties on all sides in keeping the email inboxes and phone numbers uncluttered with clueless single-residential-last-mile-user complaints about their wifi not working right.
      - jameshart 4y ago
        Right, and do those people give a crap about the fact that someone's IP space is originating some low grade HTTP noise? "Hey Bob. Bunch of clients on your network keep initiating https connections with servers on my network" "Gee Bob, sorry about that I'll go see what's causing that right away..."
        catminou 4y ago
        Some actually do. If folks are escalating to the neteng crews it usually means something is broken via normal channels and it could use attention. That's social network is exercised for a variety of things: Abuse, ddos, IP spoofing, botnet C2s being hosted somewhere, malware hosting, complex customer investigations, route leaks, route hijacks, routing loops, congestion, executives home internet issues and more.
jameshart 4y ago
Also...hang on a second. I get that morally we might feel like it is better for these ill-mannered varmints to receive a well-deserved '401 Unauthorized' HTTP response when they try their shenanigans. But... really, if they were just going to get a '404 Not found' anyway, who cares? Your server has to serve pretty much the same number of bytes, and it has to run less regexes over user agent headers. There's just something I find charmingly anachronistic about trying to operate a server like this.
- 2000UltraDeluxe 4y ago
  WordPress uses a lot of resources for 404 pages. It's dumb but WordPress is everywhere. One site being hammered is one thing. Thousands of sites being hammered at the same time is another. The issue in the article is a current and live problem. Blocking the requests saves a lot of CPU cycles.

jameshart 4y ago

Nobody’s going to argue that this is an example of a well-behaved bot. I’m just astonished that there are still people who have the time to notice and lament such behavior, and devote time to handcrafting .htaccess rules, rather than just having an aggressive set of heuristics and accepting that some crap will still get through. The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks just feels like a quaint relic of the arpanet.

robertlagrant 4y ago
> The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks just feels like a quaint relic of the arpanet. Feelings aside, it doesn't seem that different in principle to an ad blocker with a shared list of ad servers.
walrus01 4y ago
> The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks Have you ever worked in network engineering for an ISP that holds a decent sized chunk of ARIN or RIPE ipv4 IP space, and actually cares about its long term IP space reputation? There's a big difference between a $5/mo colo/VPS operator that doesn't give a damn, and an ISP that does pay attention when legit "you have customers with abusive traffic" reports reach their network engineering team. Your comment about not caring about policing badly behaved clients on a network seems like the sort of organization that would land itself on every SMTP RBL in existence. A hosting company (and Azure and AWS are just the giant corporate evolution of a hosting/colo company) needs to be prepared and have a process to fire a customer for violation of TOS/AUP. It's done all the time. The immense size and callous nature of Microsoft/Azure makes this less likely in the modern era.
- jameshart 4y ago
  SMTP abuse has its mechanisms, with widespread IP blocking mechanisms that ISPs care about. What’s the equivalent mechanism whereby abusive HTTP traffic gets identified and causes an ISP to pay attention? Is there some sort of greybeard grapevine that old school webmasters can use to get the attention of class A IP network operators?
  - walrus01 4y ago
    > Is there some sort of greybeard grapevine that old school webmasters can use to get the attention of class A IP network operators? webmasters as in single operators of httpd somewhere, no, not likely other network operators who are recognized ASNs and go to NANOG or RIPE or APNIC conferences and meet people face to face, yes. if you wilfully disregard valid human-written abuse complaints coming from other equally-clueful network engineering persons at other ISPs, you and your ISP will acquire a tainted reputation. there's various industry back-channels that exist between ISPs to communicate with each other about this sort of thing, I won't go into many details, primarily because there's a high degree of interest of the parties on all sides in keeping the email inboxes and phone numbers uncluttered with clueless single-residential-last-mile-user complaints about their wifi not working right.
    - jameshart 4y ago
      Right, and do those people give a crap about the fact that someone's IP space is originating some low grade HTTP noise? "Hey Bob. Bunch of clients on your network keep initiating https connections with servers on my network" "Gee Bob, sorry about that I'll go see what's causing that right away..."
      - catminou 4y ago
        Some actually do. If folks are escalating to the neteng crews it usually means something is broken via normal channels and it could use attention. That's social network is exercised for a variety of things: Abuse, ddos, IP spoofing, botnet C2s being hosted somewhere, malware hosting, complex customer investigations, route leaks, route hijacks, routing loops, congestion, executives home internet issues and more.

jameshart 4y ago

Also...hang on a second. I get that morally we might feel like it is better for these ill-mannered varmints to receive a well-deserved '401 Unauthorized' HTTP response when they try their shenanigans. But... really, if they were just going to get a '404 Not found' anyway, who cares? Your server has to serve pretty much the same number of bytes, and it has to run less regexes over user agent headers. There's just something I find charmingly anachronistic about trying to operate a server like this.

2000UltraDeluxe 4y ago
WordPress uses a lot of resources for 404 pages. It's dumb but WordPress is everywhere. One site being hammered is one thing. Thousands of sites being hammered at the same time is another. The issue in the article is a current and live problem. Blocking the requests saves a lot of CPU cycles.