3 ms·

The Internet is a cesspool of scams now

by reactordev 3mo ago

d1ss0nanz 3mo ago
Always was.
pocksuppet 3mo ago
scraping, however, is not intrinsically a scam.
- mplewis 3mo ago
  It is when you're doing it like the LLM companies are: at scale, to the degree that you're taking down my site, without my consent by masking your user-agent, for the purpose of stealing data I didn't authorize you to have.
  - lsaferite 3mo ago
    Unfortunately the response to that right now is to nuke everything that isn't within a strict set of constraints. That helps with the bad bulk-scrapers, but hits everyone else as well as collateral damage.
  - VorpalWay 3mo ago
    Something like Anubis in front of the server to protect it might be an option. It sucks that we have to resort to that yes, but it seems the least bad option currently (better than the entire internet going through Cloudflare at least).
    - flexagoon 3mo ago
      You can just use one line of JS instead https://fxgn.dev/blog/anubis https://fxgn.dev/blog/anubis
  - userbinator 3mo ago
    The idiotic "stealing" argument again? At least use "piracy" if you want to be correct... The moment you openly publish information on the Internet, you have already given consent. There are other solutions to bandwidth usage. User-agent discrimination should be illegal. All it does is further the control that Big Tech has, and help authoritarian governments with their control too.
    - pdpi 3mo ago
      As with anything IP-related, the data is licensed under whatever terms you choose to impose. That’s the extent of the consent you’ve provided.
      - Asraelite 3mo ago
        Permission to access is not the same as permission to use.
        NetMageSCW 3mo ago
        Reading is using.
        Asraelite 3mo ago
        No it's not. There is a legal distinction in IP law between the act of observing information and the act of using that information for some kind of personal gain.
    - matheusmoreira 3mo ago
      > User-agent discrimination should be illegal. Couldn't agree more. Nonsense like remote attestation too. If this discrimination persists, it will lead to the destruction of what little computing freedom we still have.
  - Bender 3mo ago
    I documented some crude methods that can stop most of that without a CDN. [1] There will be some false positives so I guess it depends on ones priorities which methods if any to implement or test on a throw-away test site. Not perfect, nothing is. I am watching hundreds of bots sending SYN's and the daemons are oblivious to them. The only method I have not played around with yet is #7 ssl fingerprinting. There are additional methods I chose not to document such as limiting access to logged in accounts that require double-opting-in to acceptable use policies and terms of use, not that most scrapers would give a toss. That it too much whack-a-mole for me personally. That method requires progressively adding friction to account creation and that comes with some pros and cons. [1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Some-Of-The-Bots/ https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
  - nradov 3mo ago
    What data is being stolen? Are you referring to copyright violation or something else? If you don't want LLM companies to scrape a site then just restrict access to authorized users. Simple.
    - deleted 3mo ago
      [deleted]
    - kdheiwns 3mo ago
      This is the same mindset of criminal in a low trust society btw "If you didn't want me to do this, you should had a fence/cameras/security guards. You shouldn't have dressed like that. You shouldn't have put your phone in that pocket." Excusing trillion dollar corporations like low level criminals is embarrassing. Society shouldn't have to lock itself up because bad actors are spreading everywhere. The bad actors should just be removed.
      - nradov 3mo ago
        Relax buddy, it's just HTTP requests. They're not stealing your phone.
        reactordev 3mo ago
        you never heard of HTTP request smuggling or XSS then... Let alone being able to fingerprint and dump all your browser history and session data, cookies, etc.
      - cindyllm 3mo ago
        [dead]
    - jeroenhd 3mo ago
      That's what's being done, isn't it? Fingerprinting technologies and proof of work are trying to authorize people and block bots. Unfortunately, companies like these are trying to bypass the authorization controls. If these scraping companies would respect some kind of X-Scraping-Permitted header, this whole process would be a lot better, but that's not going to happen in an industry that has already normalized using botnets.
  - parasense 3mo ago
    I think the key word was "intrinsically". It's like public photography, it's intrinsically legal, except when it's a Flock camera and then it's suddenly an invasion of privacy.
    - nativeit 3mo ago
      A Flock camera isn’t an invasion of privacy. Fleets of hundreds or thousands of Flock cameras feeding into a massive dataset is definitely an invasion of privacy, and hardly sudden.
      - tadfisher 3mo ago
        More precisely: pinpointing you at a given place and time does not need a warrant. Collecting many such pinpoints, allowing authorities to reconstruct your exact movements at some point in the future (without needing to flag you or issue a BOLO or whatever), does.
  - matheusmoreira 3mo ago
    > at scale, to the degree that you're taking down my site Fair. Scrapers should be polite and do their utmost to consume the smallest possible amount of resources. > without my consent by masking your user-agent Your consent is not required. It's my user agent. I set it to whatever I want. > for the purpose of stealing data I didn't authorize you to have Data can't be "stolen", only copied. You set up an HTTP server that literally sends people the data when they request it. Don't do that if you don't want people to have the data. Secrecy is the only possible defense.
    - lixtra 3mo ago
      > You set up an HTTP server that literally sends people the data when they request it. Don't do that if you don't want people to have the data. By the same argument I could say: If I send you an exploit and you execute it, don’t complain that your setup fell for it. Just don’t download and run random data from the internet. In reality there’s a consent and expectation beyond the pure technicals.
      - matheusmoreira 3mo ago
        > Just don’t download and run random data from the internet. I don't. I go out of my way to filter everything. Scraping is but one of the tools I use to do it. I want just the data that I actually care about, not people's javascripted hot mess websites full of malware-vectoring ads, fingerprinting and tracking. I don't let my computers talk to strangers either. My servers don't respond to just anyone, they only reply to me, and only after I've cryptographically authenticated. When others try to talk to them it's like they're not even there. But people want their computers to talk to strangers, don't they? They want to serve pages and pages of ads to massive audiences. Unlike your exploitation example, nobody's actively invading their computers and exfiltrating data. Breaking into someone else's computers and dumping their private databases is one thing. We're just requesting the exact same data that they're more than happy to send out to literally anyone who shows up with a browser, through the exact same channels even. So I really have no sympathy.
      - pocksuppet 3mo ago
  - pocksuppet 3mo ago
    That isn't the type of scraping being discussed here.
- duped 3mo ago
  No it's just intrinsically antisocial behavior
  - GaryBluto 3mo ago
    Scraping has never "intrinsically" been anti-social behaviour, you just don't like it.
Mistletoe 3mo ago
Has the internet made the average human’s life better? As someone that lived in both eras I have to say no. If you want to improve humanity greatly and prevent currently 8.3 billion people’s misery and counting, you build a time machine and prevent Tim Berners-Lee from inventing it in the first place. But it wouldn’t help, because someone else would still invent this torture nexus of hyperconnected human minds.
- matheusmoreira 3mo ago
  The internet was amazing before the big techs started centralizing everything, before governments started fragmenting it into censored regional networks...
- NetMageSCW 3mo ago
  Ask those who had no medical assistance before the Internet. Or ask Ukraine.

The Internet is a cesspool of scams now

d1ss0nanz 3mo ago

Always was.

pocksuppet 3mo ago

scraping, however, is not intrinsically a scam.

mplewis 3mo ago
It is when you're doing it like the LLM companies are: at scale, to the degree that you're taking down my site, without my consent by masking your user-agent, for the purpose of stealing data I didn't authorize you to have.
- lsaferite 3mo ago
  Unfortunately the response to that right now is to nuke everything that isn't within a strict set of constraints. That helps with the bad bulk-scrapers, but hits everyone else as well as collateral damage.
- VorpalWay 3mo ago
  Something like Anubis in front of the server to protect it might be an option. It sucks that we have to resort to that yes, but it seems the least bad option currently (better than the entire internet going through Cloudflare at least).
  - flexagoon 3mo ago
    You can just use one line of JS instead https://fxgn.dev/blog/anubis https://fxgn.dev/blog/anubis
- userbinator 3mo ago
  The idiotic "stealing" argument again? At least use "piracy" if you want to be correct... The moment you openly publish information on the Internet, you have already given consent. There are other solutions to bandwidth usage. User-agent discrimination should be illegal. All it does is further the control that Big Tech has, and help authoritarian governments with their control too.
  - pdpi 3mo ago
    As with anything IP-related, the data is licensed under whatever terms you choose to impose. That’s the extent of the consent you’ve provided.
    - Asraelite 3mo ago
      Permission to access is not the same as permission to use.
      - NetMageSCW 3mo ago
        Reading is using.
        Asraelite 3mo ago
        No it's not. There is a legal distinction in IP law between the act of observing information and the act of using that information for some kind of personal gain.
  - matheusmoreira 3mo ago
    > User-agent discrimination should be illegal. Couldn't agree more. Nonsense like remote attestation too. If this discrimination persists, it will lead to the destruction of what little computing freedom we still have.
- Bender 3mo ago
  I documented some crude methods that can stop most of that without a CDN. [1] There will be some false positives so I guess it depends on ones priorities which methods if any to implement or test on a throw-away test site. Not perfect, nothing is. I am watching hundreds of bots sending SYN's and the daemons are oblivious to them. The only method I have not played around with yet is #7 ssl fingerprinting. There are additional methods I chose not to document such as limiting access to logged in accounts that require double-opting-in to acceptable use policies and terms of use, not that most scrapers would give a toss. That it too much whack-a-mole for me personally. That method requires progressively adding friction to account creation and that comes with some pros and cons. [1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Some-Of-The-Bots/ https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
- nradov 3mo ago
  What data is being stolen? Are you referring to copyright violation or something else? If you don't want LLM companies to scrape a site then just restrict access to authorized users. Simple.
  - deleted 3mo ago
    [deleted]
  - kdheiwns 3mo ago
    This is the same mindset of criminal in a low trust society btw "If you didn't want me to do this, you should had a fence/cameras/security guards. You shouldn't have dressed like that. You shouldn't have put your phone in that pocket." Excusing trillion dollar corporations like low level criminals is embarrassing. Society shouldn't have to lock itself up because bad actors are spreading everywhere. The bad actors should just be removed.
    - nradov 3mo ago
      Relax buddy, it's just HTTP requests. They're not stealing your phone.
      - reactordev 3mo ago
        you never heard of HTTP request smuggling or XSS then... Let alone being able to fingerprint and dump all your browser history and session data, cookies, etc.
    - cindyllm 3mo ago
      [dead]
  - jeroenhd 3mo ago
    That's what's being done, isn't it? Fingerprinting technologies and proof of work are trying to authorize people and block bots. Unfortunately, companies like these are trying to bypass the authorization controls. If these scraping companies would respect some kind of X-Scraping-Permitted header, this whole process would be a lot better, but that's not going to happen in an industry that has already normalized using botnets.
- parasense 3mo ago
  I think the key word was "intrinsically". It's like public photography, it's intrinsically legal, except when it's a Flock camera and then it's suddenly an invasion of privacy.
  - nativeit 3mo ago
    A Flock camera isn’t an invasion of privacy. Fleets of hundreds or thousands of Flock cameras feeding into a massive dataset is definitely an invasion of privacy, and hardly sudden.
    - tadfisher 3mo ago
      More precisely: pinpointing you at a given place and time does not need a warrant. Collecting many such pinpoints, allowing authorities to reconstruct your exact movements at some point in the future (without needing to flag you or issue a BOLO or whatever), does.
- matheusmoreira 3mo ago
  > at scale, to the degree that you're taking down my site Fair. Scrapers should be polite and do their utmost to consume the smallest possible amount of resources. > without my consent by masking your user-agent Your consent is not required. It's my user agent. I set it to whatever I want. > for the purpose of stealing data I didn't authorize you to have Data can't be "stolen", only copied. You set up an HTTP server that literally sends people the data when they request it. Don't do that if you don't want people to have the data. Secrecy is the only possible defense.
  - lixtra 3mo ago
    > You set up an HTTP server that literally sends people the data when they request it. Don't do that if you don't want people to have the data. By the same argument I could say: If I send you an exploit and you execute it, don’t complain that your setup fell for it. Just don’t download and run random data from the internet. In reality there’s a consent and expectation beyond the pure technicals.
    - matheusmoreira 3mo ago
      > Just don’t download and run random data from the internet. I don't. I go out of my way to filter everything. Scraping is but one of the tools I use to do it. I want just the data that I actually care about, not people's javascripted hot mess websites full of malware-vectoring ads, fingerprinting and tracking. I don't let my computers talk to strangers either. My servers don't respond to just anyone, they only reply to me, and only after I've cryptographically authenticated. When others try to talk to them it's like they're not even there. But people want their computers to talk to strangers, don't they? They want to serve pages and pages of ads to massive audiences. Unlike your exploitation example, nobody's actively invading their computers and exfiltrating data. Breaking into someone else's computers and dumping their private databases is one thing. We're just requesting the exact same data that they're more than happy to send out to literally anyone who shows up with a browser, through the exact same channels even. So I really have no sympathy.
    - pocksuppet 3mo ago
- pocksuppet 3mo ago
  That isn't the type of scraping being discussed here.
duped 3mo ago
No it's just intrinsically antisocial behavior
- GaryBluto 3mo ago
  Scraping has never "intrinsically" been anti-social behaviour, you just don't like it.

Mistletoe 3mo ago

Has the internet made the average human’s life better? As someone that lived in both eras I have to say no. If you want to improve humanity greatly and prevent currently 8.3 billion people’s misery and counting, you build a time machine and prevent Tim Berners-Lee from inventing it in the first place. But it wouldn’t help, because someone else would still invent this torture nexus of hyperconnected human minds.

matheusmoreira 3mo ago
The internet was amazing before the big techs started centralizing everything, before governments started fragmenting it into censored regional networks...
NetMageSCW 3mo ago
Ask those who had no medical assistance before the Internet. Or ask Ukraine.