3 ms·

From looking at some of the rules like https://github.com/TecharoHQ/anubis/blob/main/data/bots/headless-browsers.yaml https://github.com/TecharoHQ/anubis/blob/m

From looking at some of the rules like https://github.com/TecharoHQ/anubis/blob/main/data/bots/headless-browsers.yaml https://github.com/TecharoHQ/anubis/blob/main/data/bots/head... it seems that Anubis explicitly punishes bots that are "honest" about their user agent - I might be missing something, but isn't this just pressuring anyone who does anything bot-related to just lie about their user agent? Flat out user-agent blacklist seems really weird, it's going to reward the companies that are more unethical in their scraping practices than the ones who report their user agent truthfully. From the repo it also seems like all the AI crawlers are also DENY, which, again, would reward AI companies that don't disclose their identity in the user agent.

userbinator 1y ago
User-agent header is basically useless at this point. It's trivial to set it to whatever you want, and all it does is help the browser incumbents.
- Tiberium 1y ago
  You're right, that's why I'm questioning the reason Anubis implemented it this way. Lots of big AI companies are at least honest about their crawlers and have proper user agents (which Anubis outright blocks). So "unethical" companies who change the user-agent to something normal will have an advantage with the way Anubis is currently set up by default. I'm aware that end users can modify the rules, but in reality most will just use the defaults.
  - xena 1y ago
    Shitty heuristics buy time to gather data and make better heuristics.
  - MillironX 1y ago
    Despite broadcasting their user agents properly, the AI companies ignore robots.txt and still waste my server resources. So yeah, the dishonest botnets will have an advantage, but I don't give swindlers a pass just because they rob me to my face. I'm okay with defaults that punish all bots.
    - goku12 1y ago
      You can have a bot allow list. I think it's also being planned as a subscription service (not sure about this part).
- deleted 1y ago
  [deleted]
EugeneOZ 1y ago
The point is to reduce the server load produced by bots. Honest AI scrapers use the information to learn, which increases their value, and the owner of the scraped server has to pay for it, getting nothing back — there's nothing honest about it. Search engines give you visitors, AI spiders only take your money.
wzdd 1y ago
The point of anubis is to make scraping unprofitable by forcing bots to solve a sha256-based proof-of-work captcha, so another point of view is that the explicit denylist is actually saving those bot authors time and/or money.
jeroenhd 1y ago
From what I can tell from the author's Mastodon, it seems like they're working on a fingerprinting solution to catch these fake bots in an upcoming version based on some passively observed behaviour. And, of course, the link just shows the default behaviour. Website admins can change them to their needs. I'm sure there will be workarounds (like that version of curl that has its HTTP stack replaced by Chrome's) but things are ever moving forward.

userbinator 1y ago

User-agent header is basically useless at this point. It's trivial to set it to whatever you want, and all it does is help the browser incumbents.

Tiberium 1y ago
You're right, that's why I'm questioning the reason Anubis implemented it this way. Lots of big AI companies are at least honest about their crawlers and have proper user agents (which Anubis outright blocks). So "unethical" companies who change the user-agent to something normal will have an advantage with the way Anubis is currently set up by default. I'm aware that end users can modify the rules, but in reality most will just use the defaults.
- xena 1y ago
  Shitty heuristics buy time to gather data and make better heuristics.
- MillironX 1y ago
  Despite broadcasting their user agents properly, the AI companies ignore robots.txt and still waste my server resources. So yeah, the dishonest botnets will have an advantage, but I don't give swindlers a pass just because they rob me to my face. I'm okay with defaults that punish all bots.
  - goku12 1y ago
    You can have a bot allow list. I think it's also being planned as a subscription service (not sure about this part).
deleted 1y ago
[deleted]

EugeneOZ 1y ago

The point is to reduce the server load produced by bots. Honest AI scrapers use the information to learn, which increases their value, and the owner of the scraped server has to pay for it, getting nothing back — there's nothing honest about it. Search engines give you visitors, AI spiders only take your money.

wzdd 1y ago

The point of anubis is to make scraping unprofitable by forcing bots to solve a sha256-based proof-of-work captcha, so another point of view is that the explicit denylist is actually saving those bot authors time and/or money.

jeroenhd 1y ago

From what I can tell from the author's Mastodon, it seems like they're working on a fingerprinting solution to catch these fake bots in an upcoming version based on some passively observed behaviour. And, of course, the link just shows the default behaviour. Website admins can change them to their needs. I'm sure there will be workarounds (like that version of curl that has its HTTP stack replaced by Chrome's) but things are ever moving forward.