3 ms·

I don't know if I want your traffic until I see what your traffic is. You want to look at one of our git commits? Sure! That's what our web-fronted git repo is

I don't know if I want your traffic until I see what your traffic is. You want to look at one of our git commits? Sure! That's what our web-fronted git repo is for. Go right head! Be our guest! Oh ... I see. You want to download every commit in our repository. One by one, when you have used git clone. Hmm, yeah, I don't want your traffic. But wait, "your traffic" seems to originate from ... consults fail2ban logs ... more than 900k different IP addresses, so "disconnecting" from you is non-trivial. I can't put it more politely than this: fuck off. Do not pass go. Do not collect stock options. Go to hell, and stay there.

renewiltord 1y ago
There's a protocol for that. Just reject the connection. Don't implode, just write some code. Your business model isn't my problem.
- PaulDavisThe1st 1y ago
  Reject the connection based on what? IP address (presumably after too many visits) ? So now the iptables mechanism has to scale to fit your business model (of hammering my git repository 1 commit at a time from nearly a million IP addresses) ? Why does the code I use have to fit your braindead model? We wouldn't care if you just used git clone, but you're too dumb to do that. The URL? Legitimate human (or other) users won't be happy about that. Our web-fronted git repo is not part of our business model. It's just a free service we like to offer people, unrelated to revenue flow or business operations. So your behavior is not screwing my business model, but it is screwing up people who for whatever reason want to use that service, who can no longer use the web-fronted git repo. ps. I've used "you" throughout the above because you used "my". No idea if you personally are involved in any such behavior.
- ben_w 1y ago
  Reject 900k different connections from different origins each asking for what would in isolation be fine and the only problem is the quantity?
  - Nextgrid 1y ago
    But what's the difference between one user making 900k hits and 900k different users making one hit? In both cases you have made a resource available and people are requesting it, some more than others. If serving traffic for free is a problem, don't. If you are only able to serve N requests per second/minute/day/etc, do that. But don't complain if you give out something for free and people take it. (also, a lot of the numbers people quote during these AI scraper "attacks" are very tame and the fact they are branded as problematic makes me suspect there's substantial incompetence in the solutions deployed to serve them)
    - latexr 1y ago
      > But what's the difference between one user making 900k hits and 900k different users making one hit? What’s the difference between giving 900K meals to one person and feeding 900K people? The former is being abusive, wasteful, and depriving almost 900K other people of food. They are also being deceitful by pretending to be 900K different people. Resources are finite. Web requests aren’t food, but you still pay for them. A spike in traffic may mean your service being down for the rest of the month, which is more acceptable if you helped a bunch of people who have now learned about and can talk about and share what you provided, versus having wasted all your traffic on a single bad actor who didn’t even care because they were just a robot. > makes me suspect there's substantial incompetence in the solutions deployed to serve them So you see bots scraping the Wikipedia webpages instead of downloading their organised dump, or scraping every git service webpage instead of cloning a repo, and think the incompetence is with the website instead of the scraper wasting time and resources to do a worse job?
    - PaulDavisThe1st 1y ago
      There were never 900k users interested in each commit. Never was, never will be. So that's a false comparison. These scrapers have upped both the server load (requests per second) and bandwidth requirements, without me consenting to it. If they were actual human users OR bots that were appropriately designed to minimize their impact on the target sites, that's perfectly OK. Maybe if this was truly the only way to get to our god-like LLM to work in a god-like way (*), it would also be acceptable. But it isn't. And on top of that, they are incompetently designed and they are causing real issues that a huge number of sites need to address. (*) put differently, if all this current scraping activity delivered some notable benefit to humanity
- whatevaa 1y ago
  That's exactly what they are doing. Rejecting the connection of people like you, cause you don't care. And if you start your own bussiness, you will suddenly encounter the same problem too. Then you will be able to "just write some code". Anytime somebody writes "just" you immedially can understand that they have no idea what they are talking about.
- msgodel 1y ago
  Sure. What that looks like is always using ssh to access git and things like github going away. I think most of us can agree that's probably not good. For the tools non-technical people use it's probably far worse, pretty much the end of the open web outside static personal pages. I think the ISPs serving these requests are probably going to have to start going after customers for being abusive in order for this to stop.
  - renewiltord 1y ago
    Seems fine to me. Same as ads. If you don’t want to send content with ads which I will render without ads don’t send. That ended some businesses and made others paywall. Such is life.

renewiltord 1y ago

There's a protocol for that. Just reject the connection. Don't implode, just write some code. Your business model isn't my problem.

PaulDavisThe1st 1y ago
Reject the connection based on what? IP address (presumably after too many visits) ? So now the iptables mechanism has to scale to fit your business model (of hammering my git repository 1 commit at a time from nearly a million IP addresses) ? Why does the code I use have to fit your braindead model? We wouldn't care if you just used git clone, but you're too dumb to do that. The URL? Legitimate human (or other) users won't be happy about that. Our web-fronted git repo is not part of our business model. It's just a free service we like to offer people, unrelated to revenue flow or business operations. So your behavior is not screwing my business model, but it is screwing up people who for whatever reason want to use that service, who can no longer use the web-fronted git repo. ps. I've used "you" throughout the above because you used "my". No idea if you personally are involved in any such behavior.
ben_w 1y ago
Reject 900k different connections from different origins each asking for what would in isolation be fine and the only problem is the quantity?
- Nextgrid 1y ago
  But what's the difference between one user making 900k hits and 900k different users making one hit? In both cases you have made a resource available and people are requesting it, some more than others. If serving traffic for free is a problem, don't. If you are only able to serve N requests per second/minute/day/etc, do that. But don't complain if you give out something for free and people take it. (also, a lot of the numbers people quote during these AI scraper "attacks" are very tame and the fact they are branded as problematic makes me suspect there's substantial incompetence in the solutions deployed to serve them)
  - latexr 1y ago
    > But what's the difference between one user making 900k hits and 900k different users making one hit? What’s the difference between giving 900K meals to one person and feeding 900K people? The former is being abusive, wasteful, and depriving almost 900K other people of food. They are also being deceitful by pretending to be 900K different people. Resources are finite. Web requests aren’t food, but you still pay for them. A spike in traffic may mean your service being down for the rest of the month, which is more acceptable if you helped a bunch of people who have now learned about and can talk about and share what you provided, versus having wasted all your traffic on a single bad actor who didn’t even care because they were just a robot. > makes me suspect there's substantial incompetence in the solutions deployed to serve them So you see bots scraping the Wikipedia webpages instead of downloading their organised dump, or scraping every git service webpage instead of cloning a repo, and think the incompetence is with the website instead of the scraper wasting time and resources to do a worse job?
  - PaulDavisThe1st 1y ago
    There were never 900k users interested in each commit. Never was, never will be. So that's a false comparison. These scrapers have upped both the server load (requests per second) and bandwidth requirements, without me consenting to it. If they were actual human users OR bots that were appropriately designed to minimize their impact on the target sites, that's perfectly OK. Maybe if this was truly the only way to get to our god-like LLM to work in a god-like way (*), it would also be acceptable. But it isn't. And on top of that, they are incompetently designed and they are causing real issues that a huge number of sites need to address. (*) put differently, if all this current scraping activity delivered some notable benefit to humanity
whatevaa 1y ago
That's exactly what they are doing. Rejecting the connection of people like you, cause you don't care. And if you start your own bussiness, you will suddenly encounter the same problem too. Then you will be able to "just write some code". Anytime somebody writes "just" you immedially can understand that they have no idea what they are talking about.
msgodel 1y ago
Sure. What that looks like is always using ssh to access git and things like github going away. I think most of us can agree that's probably not good. For the tools non-technical people use it's probably far worse, pretty much the end of the open web outside static personal pages. I think the ISPs serving these requests are probably going to have to start going after customers for being abusive in order for this to stop.
- renewiltord 1y ago
  Seems fine to me. Same as ads. If you don’t want to send content with ads which I will render without ads don’t send. That ended some businesses and made others paywall. Such is life.