3 ms·

If you don't want to receive data, don't. If you don't want to send data, don't. No one is asking you to receive traffic from my IPs or send to my IPs. You've j

If you don't want to receive data, don't. If you don't want to send data, don't. No one is asking you to receive traffic from my IPs or send to my IPs. You've just configured your server one way. Or to use a common HN aphorism “your business model is not my problem”. Disconnect from me if you don’t want my traffic.

PaulDavisThe1st 1y ago
I don't know if I want your traffic until I see what your traffic is. You want to look at one of our git commits? Sure! That's what our web-fronted git repo is for. Go right head! Be our guest! Oh ... I see. You want to download every commit in our repository. One by one, when you have used git clone. Hmm, yeah, I don't want your traffic. But wait, "your traffic" seems to originate from ... consults fail2ban logs ... more than 900k different IP addresses, so "disconnecting" from you is non-trivial. I can't put it more politely than this: fuck off. Do not pass go. Do not collect stock options. Go to hell, and stay there.
- renewiltord 1y ago
  There's a protocol for that. Just reject the connection. Don't implode, just write some code. Your business model isn't my problem.
  - PaulDavisThe1st 1y ago
    Reject the connection based on what? IP address (presumably after too many visits) ? So now the iptables mechanism has to scale to fit your business model (of hammering my git repository 1 commit at a time from nearly a million IP addresses) ? Why does the code I use have to fit your braindead model? We wouldn't care if you just used git clone, but you're too dumb to do that. The URL? Legitimate human (or other) users won't be happy about that. Our web-fronted git repo is not part of our business model. It's just a free service we like to offer people, unrelated to revenue flow or business operations. So your behavior is not screwing my business model, but it is screwing up people who for whatever reason want to use that service, who can no longer use the web-fronted git repo. ps. I've used "you" throughout the above because you used "my". No idea if you personally are involved in any such behavior.
  - ben_w 1y ago
    Reject 900k different connections from different origins each asking for what would in isolation be fine and the only problem is the quantity?
    - Nextgrid 1y ago
      But what's the difference between one user making 900k hits and 900k different users making one hit? In both cases you have made a resource available and people are requesting it, some more than others. If serving traffic for free is a problem, don't. If you are only able to serve N requests per second/minute/day/etc, do that. But don't complain if you give out something for free and people take it. (also, a lot of the numbers people quote during these AI scraper "attacks" are very tame and the fact they are branded as problematic makes me suspect there's substantial incompetence in the solutions deployed to serve them)
      - latexr 1y ago
        > But what's the difference between one user making 900k hits and 900k different users making one hit? What’s the difference between giving 900K meals to one person and feeding 900K people? The former is being abusive, wasteful, and depriving almost 900K other people of food. They are also being deceitful by pretending to be 900K different people. Resources are finite. Web requests aren’t food, but you still pay for them. A spike in traffic may mean your service being down for the rest of the month, which is more acceptable if you helped a bunch of people who have now learned about and can talk about and share what you provided, versus having wasted all your traffic on a single bad actor who didn’t even care because they were just a robot. > makes me suspect there's substantial incompetence in the solutions deployed to serve them So you see bots scraping the Wikipedia webpages instead of downloading their organised dump, or scraping every git service webpage instead of cloning a repo, and think the incompetence is with the website instead of the scraper wasting time and resources to do a worse job?
      - PaulDavisThe1st 1y ago
        There were never 900k users interested in each commit. Never was, never will be. So that's a false comparison. These scrapers have upped both the server load (requests per second) and bandwidth requirements, without me consenting to it. If they were actual human users OR bots that were appropriately designed to minimize their impact on the target sites, that's perfectly OK. Maybe if this was truly the only way to get to our god-like LLM to work in a god-like way (*), it would also be acceptable. But it isn't. And on top of that, they are incompetently designed and they are causing real issues that a huge number of sites need to address. (*) put differently, if all this current scraping activity delivered some notable benefit to humanity
  - whatevaa 1y ago
    That's exactly what they are doing. Rejecting the connection of people like you, cause you don't care. And if you start your own bussiness, you will suddenly encounter the same problem too. Then you will be able to "just write some code". Anytime somebody writes "just" you immedially can understand that they have no idea what they are talking about.
  - msgodel 1y ago
    Sure. What that looks like is always using ssh to access git and things like github going away. I think most of us can agree that's probably not good. For the tools non-technical people use it's probably far worse, pretty much the end of the open web outside static personal pages. I think the ISPs serving these requests are probably going to have to start going after customers for being abusive in order for this to stop.
    - renewiltord 1y ago
      Seems fine to me. Same as ads. If you don’t want to send content with ads which I will render without ads don’t send. That ended some businesses and made others paywall. Such is life.
latexr 1y ago
> Disconnect from me if you don’t want my traffic. The problem is precisely that that is not possible. It is very well known that these scrapers aren’t respecting the wishes of website owners and even circumvent blocks any way they can. If these companies respected the website owners’ desires for them to disconnect, we wouldn’t be having this conversation.
- renewiltord 1y ago
  Websites aren't people. They don't have desires. Machines have communication protocols. You can set your machine to blackhole the traffic or TCP RST or whatever you want. It's just network traffic. Do what you want with it. People send me spam. I don't whine about it. I block it.
  - latexr 1y ago
    > Websites aren't people. They don't have desires. Obviously I’m talking about the people behind them, and I very much doubt you lack the minimal mental acuity to understand that when I used “website owners” in the preceding sentence. If you don’t want to engage in a good faith discussion you can just say so, no need to waste our time with fake pedantry. But alright, I edited that section. > You can set your machine to blackhole the traffic or TCP RST or whatever you want. It's just network traffic. And then you spend all your time in a game of cat and mouse, while these scrappers bring your website down and cost you huge amounts of money. Are you incapable of understanding how that is a problem? > People send me spam. I don't whine about it. I block it. Is the amount of spam you get so overwhelming that it swamps your inbox every day to a level you’re unable to find the real messages? Do those spammers routinely circumvent your rules and filters after you’ve blocked them? Is every spam message you get costing you money? Are they increasing every day? No? Then it’s not the same thing at all.
    - whatevaa 1y ago
      I would suggest not arguing with a wall, the person you are replying to thinks there exists some magic sauce of code to solve this problem.
pjc50 1y ago
People are doing exactly that. And then other people who want to use the website are asking why they get blocked by false positives.
- renewiltord 1y ago
  Yeah, it just seems like things are playing out as one would expect them to. You're right

PaulDavisThe1st 1y ago

I don't know if I want your traffic until I see what your traffic is. You want to look at one of our git commits? Sure! That's what our web-fronted git repo is for. Go right head! Be our guest! Oh ... I see. You want to download every commit in our repository. One by one, when you have used git clone. Hmm, yeah, I don't want your traffic. But wait, "your traffic" seems to originate from ... consults fail2ban logs ... more than 900k different IP addresses, so "disconnecting" from you is non-trivial. I can't put it more politely than this: fuck off. Do not pass go. Do not collect stock options. Go to hell, and stay there.

renewiltord 1y ago
There's a protocol for that. Just reject the connection. Don't implode, just write some code. Your business model isn't my problem.
- PaulDavisThe1st 1y ago
  Reject the connection based on what? IP address (presumably after too many visits) ? So now the iptables mechanism has to scale to fit your business model (of hammering my git repository 1 commit at a time from nearly a million IP addresses) ? Why does the code I use have to fit your braindead model? We wouldn't care if you just used git clone, but you're too dumb to do that. The URL? Legitimate human (or other) users won't be happy about that. Our web-fronted git repo is not part of our business model. It's just a free service we like to offer people, unrelated to revenue flow or business operations. So your behavior is not screwing my business model, but it is screwing up people who for whatever reason want to use that service, who can no longer use the web-fronted git repo. ps. I've used "you" throughout the above because you used "my". No idea if you personally are involved in any such behavior.
- ben_w 1y ago
  Reject 900k different connections from different origins each asking for what would in isolation be fine and the only problem is the quantity?
  - Nextgrid 1y ago
    But what's the difference between one user making 900k hits and 900k different users making one hit? In both cases you have made a resource available and people are requesting it, some more than others. If serving traffic for free is a problem, don't. If you are only able to serve N requests per second/minute/day/etc, do that. But don't complain if you give out something for free and people take it. (also, a lot of the numbers people quote during these AI scraper "attacks" are very tame and the fact they are branded as problematic makes me suspect there's substantial incompetence in the solutions deployed to serve them)
    - latexr 1y ago
      > But what's the difference between one user making 900k hits and 900k different users making one hit? What’s the difference between giving 900K meals to one person and feeding 900K people? The former is being abusive, wasteful, and depriving almost 900K other people of food. They are also being deceitful by pretending to be 900K different people. Resources are finite. Web requests aren’t food, but you still pay for them. A spike in traffic may mean your service being down for the rest of the month, which is more acceptable if you helped a bunch of people who have now learned about and can talk about and share what you provided, versus having wasted all your traffic on a single bad actor who didn’t even care because they were just a robot. > makes me suspect there's substantial incompetence in the solutions deployed to serve them So you see bots scraping the Wikipedia webpages instead of downloading their organised dump, or scraping every git service webpage instead of cloning a repo, and think the incompetence is with the website instead of the scraper wasting time and resources to do a worse job?
    - PaulDavisThe1st 1y ago
      There were never 900k users interested in each commit. Never was, never will be. So that's a false comparison. These scrapers have upped both the server load (requests per second) and bandwidth requirements, without me consenting to it. If they were actual human users OR bots that were appropriately designed to minimize their impact on the target sites, that's perfectly OK. Maybe if this was truly the only way to get to our god-like LLM to work in a god-like way (*), it would also be acceptable. But it isn't. And on top of that, they are incompetently designed and they are causing real issues that a huge number of sites need to address. (*) put differently, if all this current scraping activity delivered some notable benefit to humanity
- whatevaa 1y ago
  That's exactly what they are doing. Rejecting the connection of people like you, cause you don't care. And if you start your own bussiness, you will suddenly encounter the same problem too. Then you will be able to "just write some code". Anytime somebody writes "just" you immedially can understand that they have no idea what they are talking about.
- msgodel 1y ago
  Sure. What that looks like is always using ssh to access git and things like github going away. I think most of us can agree that's probably not good. For the tools non-technical people use it's probably far worse, pretty much the end of the open web outside static personal pages. I think the ISPs serving these requests are probably going to have to start going after customers for being abusive in order for this to stop.
  - renewiltord 1y ago
    Seems fine to me. Same as ads. If you don’t want to send content with ads which I will render without ads don’t send. That ended some businesses and made others paywall. Such is life.

latexr 1y ago

> Disconnect from me if you don’t want my traffic. The problem is precisely that that is not possible. It is very well known that these scrapers aren’t respecting the wishes of website owners and even circumvent blocks any way they can. If these companies respected the website owners’ desires for them to disconnect, we wouldn’t be having this conversation.

renewiltord 1y ago
Websites aren't people. They don't have desires. Machines have communication protocols. You can set your machine to blackhole the traffic or TCP RST or whatever you want. It's just network traffic. Do what you want with it. People send me spam. I don't whine about it. I block it.
- latexr 1y ago
  > Websites aren't people. They don't have desires. Obviously I’m talking about the people behind them, and I very much doubt you lack the minimal mental acuity to understand that when I used “website owners” in the preceding sentence. If you don’t want to engage in a good faith discussion you can just say so, no need to waste our time with fake pedantry. But alright, I edited that section. > You can set your machine to blackhole the traffic or TCP RST or whatever you want. It's just network traffic. And then you spend all your time in a game of cat and mouse, while these scrappers bring your website down and cost you huge amounts of money. Are you incapable of understanding how that is a problem? > People send me spam. I don't whine about it. I block it. Is the amount of spam you get so overwhelming that it swamps your inbox every day to a level you’re unable to find the real messages? Do those spammers routinely circumvent your rules and filters after you’ve blocked them? Is every spam message you get costing you money? Are they increasing every day? No? Then it’s not the same thing at all.
  - whatevaa 1y ago
    I would suggest not arguing with a wall, the person you are replying to thinks there exists some magic sauce of code to solve this problem.

pjc50 1y ago

People are doing exactly that. And then other people who want to use the website are asking why they get blocked by false positives.

renewiltord 1y ago
Yeah, it just seems like things are playing out as one would expect them to. You're right