4 ms·

We have nothing to protect sites against scrapers except to make it more expensive for everyone’s, unless privacy-compromising or authority-trusting methods are

We have nothing to protect sites against scrapers except to make it more expensive for everyone’s, unless privacy-compromising or authority-trusting methods are on the table. Making you pay time, power, bandwidth, or money to access content does not significantly impede your browsing, so long as the cost is appropriately small. For the user above reporting thirty seconds of maxcpu, that’s excessive for a median normal person (but us hackers are not that). If giving your unique burned-in crypto-attested device ID is acceptable, there’s an entire standard for that, and when your device is found to misbehave, your device can be banned. Nintendo, Sony, Xbox call this a “console ban”; it’s quite effective because it’s stunningly expensive to replace a device. If submitting proof of citizenship through whatever attestation protocol is palatable is okay, the Anubis could simply add the digital ID web standard and let users skip the proof of work in exchange for affirming that they have a valid digital ID. But this only works if your specific identity can be banned, or else AI crawlers will just send a valid anonymized digital ID header. This problem repeats in every suggested outcome: either you make it more difficult for users to access a site, or you require users to waste energy to access a site, or you require identifiable information signed by a dependable third-party authority to be presented such that a ban is possible based on it. IP addresses don’t satisfy this; Apple IDs, trusted-vendor HSM-protected device identifiers, and digital passports do satisfy this. If you have a solution that only presents barriers to excessive use and allows abusive traffic to be revoked without depending on IP address, browser fingerprint, or paid/state credentials, then you can make billions of dollars in twelve months. Ideas welcome! This has been a problem since bots started scraping RSS feeds and republishing them as SEO blogs, and we still don’t have a solution besides Cloudflare and/or CPU-burning interstitials. (ps. I do have a solution for this, but it would require physical builds, be mildly unprofitable over time with no growth potential, and incite governments hostility towards privacy-preserving identity systems. A billionaire philanthropist could build it in a year and completely solve this problem. Sigh.)

tptacek 1y ago
We very definitely do have stuff to protect sites that don't make it more expensive for everyone! Just none of it is open source.
- necubi 1y ago
  And, fundamentally it can't be opensource. Bot detection (like anti-fraud more generally) is an adversarial game that relies on hidden techniques. Open-sourcing it means you lose that advantage and make life much easier for anyone trying to get around it.
  - tptacek 1y ago
    I think there's probably a platform for it that you can open source --- the virtual machine, or the core of the virtual machine or something, but yeah, you're right, this is something Anubis will have to contend with long term; the effective solutions for this all benefit from obscurity.
  - PaulDavisThe1st 1y ago
    There's zero reason it cannot be opensource. Proof-of-XXXXX schemes do not rely on obscurity to be functional.
    - tptacek 1y ago
      The schemes large players use to increase the cost of e.g. creating new accounts on their services do in fact rely on obscurity. They target developer cost, not compute cost.
      - PaulDavisThe1st 1y ago
        But that's not what Anubis (the subject of TFA, and most of the comment threads here) is/are about.
        tptacek 1y ago
        I think it's exactly what Anubis is about? I'm pretty familiar with what Xe is doing.
        1y ago
perching_aix 1y ago
I actually do not have a problem with digital IDs, as long as my personal identity isn't being shared alongside it. Not to the site operator, not to the government. This might seem contradictory, but I believe this is technically possible? What I don't think is this is how these solutions actually work currently. Like to basically prove that I am indeed a unique visitor who's a person according to the govt, but wouldn't reveal the person info to the site, and wouldn't reveal the site info to the govt, even if they collude. Same with the whole +18 goof. I'd actually quite like to try age gated communities, like +-5 years my age. I feel a lot of conflict stems from people coming from a bit too different walks of life sometimes. Could even do high confidence location based gating this way, which could also be cool (as well as the exact opposite of cool, because of course).
- fluoridation 1y ago
  Assuming a person can only have a single ID, how would that be enforced without a unique party having a 1-to-1 mapping between person and ID?
  - perching_aix 1y ago
    The party to hold a gov id - site id mapping would be you. The rest can then be facilitated using zero-knowledge proofs, I believe. I'm not super well-versed in crypto though, so I confess this is a lot more conjecture than knowledge.
    - fluoridation 1y ago
      So the user generates the ID for each site he visits? What prevents them from generating arbitrary IDs? The only way I can imagine this working is: 1. You go to the government and request to have a digital ID generated. 2. The government generates a random number. 3. The government issues a request to an NGO to generate a new cryptographic object based on the random number, and receives back a retrieval number. 4. The government gives you the retrieval number, which you can use to get your digital ID from the NGO. This way, the government only has the mapping between your identity and a random number, and the NGO only has the mapping between the random number and the generated object, with no possibility to deanonymize it because you don't present any ID to get it. Obviously, there must be no information exchange between the government and the NGO.
PaulDavisThe1st 1y ago
> so long as the cost is appropriately small. there are different metrics for cost, however. Based on cpu utilization and/or time, it's hard to argue that Anubis is a high price. But if it is important to you to not run javascript for whatever reason, the price of access to a site using Anubis is rather high.
userbinator 1y ago
No, no, no, hell fucking no!! You put stuff on the public Internet, expect it to be read by everyone. Don't like that? Put it behind a login. How did the propaganda persuade people into accepting mass surveillance and normalising the invasion of privacy for something that was never really a problem? I suspect the ones pushing this are the ones running these "crawlers" themselves, much like Cloudflare hosts providers of DDoS services. What a coincidence that "identity verification" became a hot topic recently. Don't even try to solve this nonexistent "problem", because that will only drive us further into authoritarian technocracy.
- altairprime 1y ago
  > What a coincidence that "identity verification" became a hot topic recently. Crying “Conspiracy” in reply to a career Chicken Little is comedic. I’ve been raising warnings about identity verification looming on the horizon for perhaps fifteen years now; thanks to DejaNews for that early realization, I suppose. > Don't even try to solve this nonexistent "problem", because that will only drive us further into authoritarian technocracy. I would celebrate and tell all my friends if someone on this thread, on any thread, would explain how we solve this without bankrupting non-business site operators and without a third-party authority. Anubis is a band-aid at best, yet no better solution — not even an idea — is presented alongside your objections. > You put stuff on the public Internet, expect it to be read by everyone. My hobbyist forum can barely stay online eight hours a day due to crawler traffic. Someone scraped the entire site by spawning one request per page with no fork limit last year. It was down for a solid week after that, and now has very severe limits in place. I don’t know how they can afford to stay running, but certainly “static only” isn’t going to solve the CPU and bandwidth costs incurred by incompetent and redundant AI crawlers. So, by making their site public in today’s infested internet, their content is no longer accessible. > Don't like that? Put it behind a login. As I noted above, one solution is payment — since free credentials registration is not an obstacle to AI bots, after all. For some reason people don’t like to charge money for hobbyist content if they can avoid it. I recognize why and am trying my best to discover a non-monetary solution on their behalf. > I suspect the ones pushing this are the ones running these "crawlers" themselves, much like Cloudflare hosts providers of DDoS services. I do not, have not, and will not run crawlers or AI agents, trainers, or other such shit at any time in the past thirty years and will continue to abstain from the entire category, which should be quite easy as I’m a retired sysop now attending full-time accounting school and giving a finger to the entire industry to pursue work that benefits humanity. Same reason I bother telling HN “the anonymity sky is falling” every so often: I’d much prefer it if we didn’t have to sacrifice anonymity online to defeat scraper bots. > No, no, no, hell fucking no!! Please find a way to turn your vehemence and passion into a productive contribution, before it’s too late for all of us. As presented, your argument is neither supported nor persuasive, and your hostility only gives opponents of anonymity more arrows in their quiver to shoot at us.
motorest 1y ago
> We have nothing to protect sites against scrapers except to make it more expensive for everyone’s (...) Authentication works, doesn't it?
- altairprime 1y ago
  Paid credentials work, yes. Free credentials temporarily do, but only until someone teachers crawlers to AI-automate signups. Forum spammers already figured this out a long time ago. A while back someone from Financial Times commented on HN about how it was confusing that we were all so hostile to their free registration required article paywall. The HN community view on their registration requirement suggests that free authentication does not work and has not for some time; has that viewpoint changed in recent years?
  - motorest 1y ago
    > Free credentials temporarily do, but only until someone teachers crawlers to AI-automate signups. Forum spammers already figured this out a long time ago. Forum spamming is a solved problem, isn't it? I mean, do you see any spam here on HN? > The HN community view on their registration requirement suggests that free authentication does not work and has not for some time; has that viewpoint changed in recent years? I don't think there is such a thing as a "HN community view". Are there meetings to agree on a consensus on important topics? Or were you trying to pull an appeal to authority?
    - altairprime 1y ago
      > Forum spamming is a solved problem, isn't it? I mean, do you see any spam here on HN? Yes, I’ve reported two or three this past week or so by emailing the mods. Do you browse /new often? What do you think makes HN a spam-free place? > I don't think there is such a thing as a "HN community view". Noted.
o11c 1y ago
SQRL solves at least part of the ID problem.

tptacek 1y ago

We very definitely do have stuff to protect sites that don't make it more expensive for everyone! Just none of it is open source.

necubi 1y ago
And, fundamentally it can't be opensource. Bot detection (like anti-fraud more generally) is an adversarial game that relies on hidden techniques. Open-sourcing it means you lose that advantage and make life much easier for anyone trying to get around it.
- tptacek 1y ago
  I think there's probably a platform for it that you can open source --- the virtual machine, or the core of the virtual machine or something, but yeah, you're right, this is something Anubis will have to contend with long term; the effective solutions for this all benefit from obscurity.
- PaulDavisThe1st 1y ago
  There's zero reason it cannot be opensource. Proof-of-XXXXX schemes do not rely on obscurity to be functional.
  - tptacek 1y ago
    The schemes large players use to increase the cost of e.g. creating new accounts on their services do in fact rely on obscurity. They target developer cost, not compute cost.
    - PaulDavisThe1st 1y ago
      But that's not what Anubis (the subject of TFA, and most of the comment threads here) is/are about.
      - tptacek 1y ago
        I think it's exactly what Anubis is about? I'm pretty familiar with what Xe is doing.
        1y ago

perching_aix 1y ago

I actually do not have a problem with digital IDs, as long as my personal identity isn't being shared alongside it. Not to the site operator, not to the government. This might seem contradictory, but I believe this is technically possible? What I don't think is this is how these solutions actually work currently. Like to basically prove that I am indeed a unique visitor who's a person according to the govt, but wouldn't reveal the person info to the site, and wouldn't reveal the site info to the govt, even if they collude. Same with the whole +18 goof. I'd actually quite like to try age gated communities, like +-5 years my age. I feel a lot of conflict stems from people coming from a bit too different walks of life sometimes. Could even do high confidence location based gating this way, which could also be cool (as well as the exact opposite of cool, because of course).

fluoridation 1y ago
Assuming a person can only have a single ID, how would that be enforced without a unique party having a 1-to-1 mapping between person and ID?
- perching_aix 1y ago
  The party to hold a gov id - site id mapping would be you. The rest can then be facilitated using zero-knowledge proofs, I believe. I'm not super well-versed in crypto though, so I confess this is a lot more conjecture than knowledge.
  - fluoridation 1y ago
    So the user generates the ID for each site he visits? What prevents them from generating arbitrary IDs? The only way I can imagine this working is: 1. You go to the government and request to have a digital ID generated. 2. The government generates a random number. 3. The government issues a request to an NGO to generate a new cryptographic object based on the random number, and receives back a retrieval number. 4. The government gives you the retrieval number, which you can use to get your digital ID from the NGO. This way, the government only has the mapping between your identity and a random number, and the NGO only has the mapping between the random number and the generated object, with no possibility to deanonymize it because you don't present any ID to get it. Obviously, there must be no information exchange between the government and the NGO.

PaulDavisThe1st 1y ago

> so long as the cost is appropriately small. there are different metrics for cost, however. Based on cpu utilization and/or time, it's hard to argue that Anubis is a high price. But if it is important to you to not run javascript for whatever reason, the price of access to a site using Anubis is rather high.

userbinator 1y ago

No, no, no, hell fucking no!! You put stuff on the public Internet, expect it to be read by everyone. Don't like that? Put it behind a login. How did the propaganda persuade people into accepting mass surveillance and normalising the invasion of privacy for something that was never really a problem? I suspect the ones pushing this are the ones running these "crawlers" themselves, much like Cloudflare hosts providers of DDoS services. What a coincidence that "identity verification" became a hot topic recently. Don't even try to solve this nonexistent "problem", because that will only drive us further into authoritarian technocracy.

altairprime 1y ago
> What a coincidence that "identity verification" became a hot topic recently. Crying “Conspiracy” in reply to a career Chicken Little is comedic. I’ve been raising warnings about identity verification looming on the horizon for perhaps fifteen years now; thanks to DejaNews for that early realization, I suppose. > Don't even try to solve this nonexistent "problem", because that will only drive us further into authoritarian technocracy. I would celebrate and tell all my friends if someone on this thread, on any thread, would explain how we solve this without bankrupting non-business site operators and without a third-party authority. Anubis is a band-aid at best, yet no better solution — not even an idea — is presented alongside your objections. > You put stuff on the public Internet, expect it to be read by everyone. My hobbyist forum can barely stay online eight hours a day due to crawler traffic. Someone scraped the entire site by spawning one request per page with no fork limit last year. It was down for a solid week after that, and now has very severe limits in place. I don’t know how they can afford to stay running, but certainly “static only” isn’t going to solve the CPU and bandwidth costs incurred by incompetent and redundant AI crawlers. So, by making their site public in today’s infested internet, their content is no longer accessible. > Don't like that? Put it behind a login. As I noted above, one solution is payment — since free credentials registration is not an obstacle to AI bots, after all. For some reason people don’t like to charge money for hobbyist content if they can avoid it. I recognize why and am trying my best to discover a non-monetary solution on their behalf. > I suspect the ones pushing this are the ones running these "crawlers" themselves, much like Cloudflare hosts providers of DDoS services. I do not, have not, and will not run crawlers or AI agents, trainers, or other such shit at any time in the past thirty years and will continue to abstain from the entire category, which should be quite easy as I’m a retired sysop now attending full-time accounting school and giving a finger to the entire industry to pursue work that benefits humanity. Same reason I bother telling HN “the anonymity sky is falling” every so often: I’d much prefer it if we didn’t have to sacrifice anonymity online to defeat scraper bots. > No, no, no, hell fucking no!! Please find a way to turn your vehemence and passion into a productive contribution, before it’s too late for all of us. As presented, your argument is neither supported nor persuasive, and your hostility only gives opponents of anonymity more arrows in their quiver to shoot at us.

motorest 1y ago

> We have nothing to protect sites against scrapers except to make it more expensive for everyone’s (...) Authentication works, doesn't it?

altairprime 1y ago
Paid credentials work, yes. Free credentials temporarily do, but only until someone teachers crawlers to AI-automate signups. Forum spammers already figured this out a long time ago. A while back someone from Financial Times commented on HN about how it was confusing that we were all so hostile to their free registration required article paywall. The HN community view on their registration requirement suggests that free authentication does not work and has not for some time; has that viewpoint changed in recent years?
- motorest 1y ago
  > Free credentials temporarily do, but only until someone teachers crawlers to AI-automate signups. Forum spammers already figured this out a long time ago. Forum spamming is a solved problem, isn't it? I mean, do you see any spam here on HN? > The HN community view on their registration requirement suggests that free authentication does not work and has not for some time; has that viewpoint changed in recent years? I don't think there is such a thing as a "HN community view". Are there meetings to agree on a consensus on important topics? Or were you trying to pull an appeal to authority?
  - altairprime 1y ago
    > Forum spamming is a solved problem, isn't it? I mean, do you see any spam here on HN? Yes, I’ve reported two or three this past week or so by emailing the mods. Do you browse /new often? What do you think makes HN a spam-free place? > I don't think there is such a thing as a "HN community view". Noted.

o11c 1y ago

SQRL solves at least part of the ID problem.