5 ms·

Insane, I wonder if we eventually end up with a non-search-engine indexed version of the web that's more like browsing in the 90s where websites just had to lin

Insane, I wonder if we eventually end up with a non-search-engine indexed version of the web that's more like browsing in the 90s where websites just had to link to oneanother to get noticed . . . . I love that the solution to LLM scraping is to serve the browser a proof of work, before they allow access - I wonder if things like new sites start to do this . . . It would mean they won't be indexed by search engines, but it would help to protect the IP

xena 2y ago
Hi! I do this! See https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis for more info!
- greybox 2y ago
  thank you for your contribution to society!
- hmry 2y ago
  I hope lots of websites adopt this, mainly because I want to see more happy jackal girls while browsing.
  - xena 2y ago
    My monetization strategy is unironically to offer a de-anime'd version under the name Techaro BotStopper or something.
    - 20after4 2y ago
      It's a clever (and hilarious) strategy that will probably sell at least a few licenses. As an anime hater I'd be motivated by this.
      - floren 2y ago
        Just change the pictures in cmd/anubis/static/img/ to whatever you prefer, I think.
        ozornin 2y ago
        Or, alternatively, you know, pay the author for the work they've done
        floren 2y ago
        Sure, if you're going to deploy it on your company site, but I think if you're running a personal website and want to throttle LLM crawlers without falsely advertising that you're a furry, you could just go and modify this piece of MIT-licensed software.
        xena 2y ago
        Or you could pay for BotStopper and have RPM/DEB packages too.
        numpad0 2y ago
        Or, alternatively, just embrace anime-porn content spectrum. I mean, just compare platforms that are free of it and ones that are chock full, and see which ones die and which grows.
    - pipe01 2y ago
      That's kinda what the company Friendly Captcha does
- noveltyaccount 2y ago
  Does the PoW make money via crypto mining? Or is it just to waste the caller's CPU cycles? If you could monetize the PoW then you could re-challenge at an interval tuned so that the caller pays for their usage.
  - xena 2y ago
    It's to waste CPU cycles. I don't want to touch cryptocurrency with a 20 foot pole. I realize I'm leaving money on the table by doing this, but I don't want to alienate the kinds of communities I want to protect.
    - dematz 2y ago
      I always felt like crypto is nothing but speculating on value with no other good uses, but there is a kind of motivation here. Say a hash challenge gets widely adopted, and scraping becomes more costly, maybe even requires GPUs. This is great, you can declare victory. But what if after a while the scraping companies, with more resources than real users, are better able to solve the hash? Crypto appeals here because you could make the scrapers cover the cost of serving their page. Ofc if you’re leery of crypto you could try to find something else for bots to do. xkcd 810 for example. Or folding at home or something. But something to make the bot traffic productive, because if it’s just a hardware capability check maybe the scrapers get better hardware than the real users. Or not, no clue idk
    - dpc_01234 2y ago
      By doing PoW as a side effect of something you need to do anyway for other reason, you actually make mining less profitable for other miners, which is helping to eliminate waste. This is an aspect that a lot of PoW haters miss. While PoW is a waste, there are long term economic incentives to minimize it to either being a side-effect of something actually useful, or using energy that would go to waste anyway, making it's overall effect gravitate toward neutral. Unfortunately such a second order effects are hard to explain to most people.
  - joquarky 2y ago
    This feels like one of the few ways to potentially avoid what seems like the inevitability of attestation.

xena 2y ago

Hi! I do this! See https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis for more info!

greybox 2y ago
thank you for your contribution to society!
hmry 2y ago
I hope lots of websites adopt this, mainly because I want to see more happy jackal girls while browsing.
- xena 2y ago
  My monetization strategy is unironically to offer a de-anime'd version under the name Techaro BotStopper or something.
  - 20after4 2y ago
    It's a clever (and hilarious) strategy that will probably sell at least a few licenses. As an anime hater I'd be motivated by this.
    - floren 2y ago
      Just change the pictures in cmd/anubis/static/img/ to whatever you prefer, I think.
      - ozornin 2y ago
        Or, alternatively, you know, pay the author for the work they've done
        floren 2y ago
        Sure, if you're going to deploy it on your company site, but I think if you're running a personal website and want to throttle LLM crawlers without falsely advertising that you're a furry, you could just go and modify this piece of MIT-licensed software.
        xena 2y ago
        Or you could pay for BotStopper and have RPM/DEB packages too.
        numpad0 2y ago
        Or, alternatively, just embrace anime-porn content spectrum. I mean, just compare platforms that are free of it and ones that are chock full, and see which ones die and which grows.
  - pipe01 2y ago
    That's kinda what the company Friendly Captcha does
noveltyaccount 2y ago
Does the PoW make money via crypto mining? Or is it just to waste the caller's CPU cycles? If you could monetize the PoW then you could re-challenge at an interval tuned so that the caller pays for their usage.
- xena 2y ago
  It's to waste CPU cycles. I don't want to touch cryptocurrency with a 20 foot pole. I realize I'm leaving money on the table by doing this, but I don't want to alienate the kinds of communities I want to protect.
  - dematz 2y ago
    I always felt like crypto is nothing but speculating on value with no other good uses, but there is a kind of motivation here. Say a hash challenge gets widely adopted, and scraping becomes more costly, maybe even requires GPUs. This is great, you can declare victory. But what if after a while the scraping companies, with more resources than real users, are better able to solve the hash? Crypto appeals here because you could make the scrapers cover the cost of serving their page. Ofc if you’re leery of crypto you could try to find something else for bots to do. xkcd 810 for example. Or folding at home or something. But something to make the bot traffic productive, because if it’s just a hardware capability check maybe the scrapers get better hardware than the real users. Or not, no clue idk
  - dpc_01234 2y ago
    By doing PoW as a side effect of something you need to do anyway for other reason, you actually make mining less profitable for other miners, which is helping to eliminate waste. This is an aspect that a lot of PoW haters miss. While PoW is a waste, there are long term economic incentives to minimize it to either being a side-effect of something actually useful, or using energy that would go to waste anyway, making it's overall effect gravitate toward neutral. Unfortunately such a second order effects are hard to explain to most people.
- joquarky 2y ago
  This feels like one of the few ways to potentially avoid what seems like the inevitability of attestation.