3 ms·

Oh my goodness - this looks to be an actual surviving webmaster, still living and almost unchanged and somehow still thriving on the modern internet. It was tho

Oh my goodness - this looks to be an actual surviving webmaster, still living and almost unchanged and somehow still thriving on the modern internet. It was thought that they were all extinct! From an anthropological point of view one of the most fascinating behaviors here is the apparent belief that as a website operator, your opinions about how HTTP clients should craft their user agent headers have any weight whatsoever. This curious belief system predates even the browser wars let alone the modern web. Remarkable to find an original example of this idealistic world view still being expounded today.

muppetman 4y ago
One hopes we can get them into captivity to start a breeding programme.
walrus01 4y ago
I get your point about most people ignoring the absolute shit flood of spurious internet traffic these days. But I don't think anyone can argue that it's the right thing to do to run a large scale web crawler/indexer with a fake user agent. Google and Bing and others are reasonably good citizens about this. Of course anyone who wants to can set up something that ignores a robots.txt file and leave it as the problem of the httpd operator to either accept or block their traffic. Not caring about false useragents is one thing, caring about somebody running a multi-thousand-requests-per-minute spider requesting pages that no longer exist is another thing. It would still be valid cause for complaint even if it had a valid useragent. The internet would be a better place if more experienced people (such as those who were running httpd 22-24 years ago) reinforced and emphasized the principle, when training and mentoring younger people, that being a good citizen with your network traffic is the responsible and ethical thing to do.
- jameshart 4y ago
  Nobody’s going to argue that this is an example of a well-behaved bot. I’m just astonished that there are still people who have the time to notice and lament such behavior, and devote time to handcrafting .htaccess rules, rather than just having an aggressive set of heuristics and accepting that some crap will still get through. The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks just feels like a quaint relic of the arpanet.
  - robertlagrant 4y ago
    > The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks just feels like a quaint relic of the arpanet. Feelings aside, it doesn't seem that different in principle to an ad blocker with a shared list of ad servers.
  - walrus01 4y ago
    > The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks Have you ever worked in network engineering for an ISP that holds a decent sized chunk of ARIN or RIPE ipv4 IP space, and actually cares about its long term IP space reputation? There's a big difference between a $5/mo colo/VPS operator that doesn't give a damn, and an ISP that does pay attention when legit "you have customers with abusive traffic" reports reach their network engineering team. Your comment about not caring about policing badly behaved clients on a network seems like the sort of organization that would land itself on every SMTP RBL in existence. A hosting company (and Azure and AWS are just the giant corporate evolution of a hosting/colo company) needs to be prepared and have a process to fire a customer for violation of TOS/AUP. It's done all the time. The immense size and callous nature of Microsoft/Azure makes this less likely in the modern era.
    - jameshart 4y ago
      SMTP abuse has its mechanisms, with widespread IP blocking mechanisms that ISPs care about. What’s the equivalent mechanism whereby abusive HTTP traffic gets identified and causes an ISP to pay attention? Is there some sort of greybeard grapevine that old school webmasters can use to get the attention of class A IP network operators?
      - walrus01 4y ago
        > Is there some sort of greybeard grapevine that old school webmasters can use to get the attention of class A IP network operators? webmasters as in single operators of httpd somewhere, no, not likely other network operators who are recognized ASNs and go to NANOG or RIPE or APNIC conferences and meet people face to face, yes. if you wilfully disregard valid human-written abuse complaints coming from other equally-clueful network engineering persons at other ISPs, you and your ISP will acquire a tainted reputation. there's various industry back-channels that exist between ISPs to communicate with each other about this sort of thing, I won't go into many details, primarily because there's a high degree of interest of the parties on all sides in keeping the email inboxes and phone numbers uncluttered with clueless single-residential-last-mile-user complaints about their wifi not working right.
- jameshart 4y ago
  Also...hang on a second. I get that morally we might feel like it is better for these ill-mannered varmints to receive a well-deserved '401 Unauthorized' HTTP response when they try their shenanigans. But... really, if they were just going to get a '404 Not found' anyway, who cares? Your server has to serve pretty much the same number of bytes, and it has to run less regexes over user agent headers. There's just something I find charmingly anachronistic about trying to operate a server like this.
  - 2000UltraDeluxe 4y ago
    WordPress uses a lot of resources for 404 pages. It's dumb but WordPress is everywhere. One site being hammered is one thing. Thousands of sites being hammered at the same time is another. The issue in the article is a current and live problem. Blocking the requests saves a lot of CPU cycles.
deleted 4y ago
[deleted]
wankle 4y ago
Such stoic, staid prose could be worthy of an all knowing Microsoft MVP status.

muppetman 4y ago

One hopes we can get them into captivity to start a breeding programme.

walrus01 4y ago

I get your point about most people ignoring the absolute shit flood of spurious internet traffic these days. But I don't think anyone can argue that it's the right thing to do to run a large scale web crawler/indexer with a fake user agent. Google and Bing and others are reasonably good citizens about this. Of course anyone who wants to can set up something that ignores a robots.txt file and leave it as the problem of the httpd operator to either accept or block their traffic. Not caring about false useragents is one thing, caring about somebody running a multi-thousand-requests-per-minute spider requesting pages that no longer exist is another thing. It would still be valid cause for complaint even if it had a valid useragent. The internet would be a better place if more experienced people (such as those who were running httpd 22-24 years ago) reinforced and emphasized the principle, when training and mentoring younger people, that being a good citizen with your network traffic is the responsible and ethical thing to do.

jameshart 4y ago
Nobody’s going to argue that this is an example of a well-behaved bot. I’m just astonished that there are still people who have the time to notice and lament such behavior, and devote time to handcrafting .htaccess rules, rather than just having an aggressive set of heuristics and accepting that some crap will still get through. The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks just feels like a quaint relic of the arpanet.
- robertlagrant 4y ago
  > The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks just feels like a quaint relic of the arpanet. Feelings aside, it doesn't seem that different in principle to an ad blocker with a shared list of ad servers.
- walrus01 4y ago
  > The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks Have you ever worked in network engineering for an ISP that holds a decent sized chunk of ARIN or RIPE ipv4 IP space, and actually cares about its long term IP space reputation? There's a big difference between a $5/mo colo/VPS operator that doesn't give a damn, and an ISP that does pay attention when legit "you have customers with abusive traffic" reports reach their network engineering team. Your comment about not caring about policing badly behaved clients on a network seems like the sort of organization that would land itself on every SMTP RBL in existence. A hosting company (and Azure and AWS are just the giant corporate evolution of a hosting/colo company) needs to be prepared and have a process to fire a customer for violation of TOS/AUP. It's done all the time. The immense size and callous nature of Microsoft/Azure makes this less likely in the modern era.
  - jameshart 4y ago
    SMTP abuse has its mechanisms, with widespread IP blocking mechanisms that ISPs care about. What’s the equivalent mechanism whereby abusive HTTP traffic gets identified and causes an ISP to pay attention? Is there some sort of greybeard grapevine that old school webmasters can use to get the attention of class A IP network operators?
    - walrus01 4y ago
      > Is there some sort of greybeard grapevine that old school webmasters can use to get the attention of class A IP network operators? webmasters as in single operators of httpd somewhere, no, not likely other network operators who are recognized ASNs and go to NANOG or RIPE or APNIC conferences and meet people face to face, yes. if you wilfully disregard valid human-written abuse complaints coming from other equally-clueful network engineering persons at other ISPs, you and your ISP will acquire a tainted reputation. there's various industry back-channels that exist between ISPs to communicate with each other about this sort of thing, I won't go into many details, primarily because there's a high degree of interest of the parties on all sides in keeping the email inboxes and phone numbers uncluttered with clueless single-residential-last-mile-user complaints about their wifi not working right.
jameshart 4y ago
Also...hang on a second. I get that morally we might feel like it is better for these ill-mannered varmints to receive a well-deserved '401 Unauthorized' HTTP response when they try their shenanigans. But... really, if they were just going to get a '404 Not found' anyway, who cares? Your server has to serve pretty much the same number of bytes, and it has to run less regexes over user agent headers. There's just something I find charmingly anachronistic about trying to operate a server like this.
- 2000UltraDeluxe 4y ago
  WordPress uses a lot of resources for 404 pages. It's dumb but WordPress is everywhere. One site being hammered is one thing. Thousands of sites being hammered at the same time is another. The issue in the article is a current and live problem. Blocking the requests saves a lot of CPU cycles.

deleted 4y ago

[deleted]

wankle 4y ago

Such stoic, staid prose could be worthy of an all knowing Microsoft MVP status.