4 ms·

tirreno (1) guy here. Our open-source system can block IP addresses based on rules triggered by specific behavior. Can you elaborate on what exact type of cra

tirreno (1) guy here. Our open-source system can block IP addresses based on rules triggered by specific behavior. Can you elaborate on what exact type of crawlers you would like to block? Like, a leaky bucket of a certain number of requests per minute? 1. https://github.com/tirrenotechnologies/tirreno https://github.com/tirrenotechnologies/tirreno

notachatbot123 10mo ago
The article is about AI web crawlers. How can your tool help and how would one set it up for this specific context?
- reconnecting 10mo ago
  I don't see how an AI crawler is different from any others. The simplest approach is to count the UA as risky or flag multiple 404 errors or HEAD requests, and block on that. Those are rules we already have out of the box. It's open source, there's no pain in writing specific rules for rate limiting, thus my question. Plus, we have developed a dashboard for manually choosing UA blocks based on name, but we're still not sure if this is something that would be really helpful for website operators.
  - Roark66 10mo ago
    >It's open source, there's no pain in writing specific rules for rate limiting, thus my question. Depends on the goal. Author wants his instance not to get killed. Request rate limiting may achieve that easily in a way transparent to normal users.
  - mmarian 10mo ago
    > count the UA as risky It's trivial to spoof UAs unfortunately.
    - reconnecting 10mo ago
      It depends. If you want to stop OAI-SearchBot/1.3, UA will be enough.
      - mmarian 10mo ago
        Why would you need tirreno if you just want to stop OAI's bot though?
        reconnecting 10mo ago
        OAI's is just an example that's easy to explain. I believe that if something is publicly available, it shouldn't be overprotected in most cases. However, there are many advanced cases, such as crawlers that collect data for platform impersonation (for scams) or custom phishing attacks, or account brute-force attacks. In those cases, I use tirreno to understand traffic through different dimensions.
reconnecting 10mo ago
I believe there is a slight misunderstanding regarding the role of 'AI crawlers'. Bad crawlers have been there since the very beginning. Some of them looking for known vulnerabilities, some scraping content for third-party services. Most of them have spoofed UAs to pretend to be legitimate bots. This is approximately 30–50% of traffic on any website.
mmarian 10mo ago
> block IP addresses based on rules triggered by specific behavior Problem is, bots can easily can resort to resi proxies, at which point you'll end up blocking legitimate traffic.
- reconnecting 10mo ago
  Again, it depends. Residential proxies are much more expensive, and most vulnerability scanners will never shift to them. I believe that there is a low chance that a real customer behind this residential IP will come to your resource. If you do an EU service, there is no pain to block Asian IPs and vice-versa. What is really important here is that most people block IPs on autopilot without seeing the distribution of their actions, and this really matters.

notachatbot123 10mo ago

The article is about AI web crawlers. How can your tool help and how would one set it up for this specific context?

reconnecting 10mo ago
I don't see how an AI crawler is different from any others. The simplest approach is to count the UA as risky or flag multiple 404 errors or HEAD requests, and block on that. Those are rules we already have out of the box. It's open source, there's no pain in writing specific rules for rate limiting, thus my question. Plus, we have developed a dashboard for manually choosing UA blocks based on name, but we're still not sure if this is something that would be really helpful for website operators.
- Roark66 10mo ago
  >It's open source, there's no pain in writing specific rules for rate limiting, thus my question. Depends on the goal. Author wants his instance not to get killed. Request rate limiting may achieve that easily in a way transparent to normal users.
- mmarian 10mo ago
  > count the UA as risky It's trivial to spoof UAs unfortunately.
  - reconnecting 10mo ago
    It depends. If you want to stop OAI-SearchBot/1.3, UA will be enough.
    - mmarian 10mo ago
      Why would you need tirreno if you just want to stop OAI's bot though?
      - reconnecting 10mo ago
        OAI's is just an example that's easy to explain. I believe that if something is publicly available, it shouldn't be overprotected in most cases. However, there are many advanced cases, such as crawlers that collect data for platform impersonation (for scams) or custom phishing attacks, or account brute-force attacks. In those cases, I use tirreno to understand traffic through different dimensions.

reconnecting 10mo ago

I believe there is a slight misunderstanding regarding the role of 'AI crawlers'. Bad crawlers have been there since the very beginning. Some of them looking for known vulnerabilities, some scraping content for third-party services. Most of them have spoofed UAs to pretend to be legitimate bots. This is approximately 30–50% of traffic on any website.

mmarian 10mo ago

> block IP addresses based on rules triggered by specific behavior Problem is, bots can easily can resort to resi proxies, at which point you'll end up blocking legitimate traffic.

reconnecting 10mo ago
Again, it depends. Residential proxies are much more expensive, and most vulnerability scanners will never shift to them. I believe that there is a low chance that a real customer behind this residential IP will come to your resource. If you do an EU service, there is no pain to block Asian IPs and vice-versa. What is really important here is that most people block IPs on autopilot without seeing the distribution of their actions, and this really matters.