4 ms·

Fantastic work by Xe here -- not the first but this seems like the most traction I've seen on a PoW anti-scraper project (with an MIT license to boot!). PoW an

Fantastic work by Xe here -- not the first but this seems like the most traction I've seen on a PoW anti-scraper project (with an MIT license to boot!). PoW anti-scraper tools are a good first step, but why don't we just jump straight to the endgame? We're drawing closer to a point where information's value is actually fully realized -- people will stop sharing knowledge for free. It doesn't have to be that way, but it does in a world where people are pressed for economic means, knowledge becomes an obvious thing to convert to capital and attempt to extract rent on. The simple way this happens is just a login wall -- for every website. It doesn't have to be a paid login wall of course (at first), but it's a super simple way to both legally and practically protect from scrapers. I think high quality knowledge, source code (which is basically executable knowledge), being open in general is a miracle/luxury of functioning, economically balanced societies where people feel driven (for many possible reasons) to give back, or have time to think of more than surviving. Don't get me wrong -- the doomer angle is almost always wrong -- every year humanity is almost always better off than we were the previous year on many important metrics, but it's getting harder to see a world where we cartwheel through another technological transformation that this time could possibly impact large percentages of the working population.

g-b-r 1y ago
Yes, let's turn the whole web into Facebook, what a bright future
- Schiendelman 1y ago
  I think we only have two choices here: 1) every webpage requires Facebook login, and then Facebook offers free hosting for the content. 2) every webpage requires some other method of login, but not locked into a single system. I read the GP comment as suggesting we push on the second option there rather than passively waiting for the first option.
  - g-b-r 1y ago
    I hope you see that you shatter anonymity and the open web with that, single system or not
    - Schiendelman 1y ago
      Of course I do. But that's already gone.
      - g-b-r 1y ago
        You must be on a different internet than mine
        Schiendelman 1y ago
        I'm not. But what are you claiming to be true?
    - hardwaresofton 1y ago
      anonymity and the open web are different things, and neither of them were promised/guaranteed to anyone on the internet. For people that value anonymity, they'll create their own spaces. People that value openness will continue to be open. What we're about to find out is what happens when the tide goes out and people show you what they really believe/want -- anything other than that is a form of social control, whether via browbeating or other means.
      - g-b-r 1y ago
        > anonymity and the open web are different things, and neither of them were promised/guaranteed to anyone on the internet. > > For people that value anonymity, they'll create their own spaces. People that value openness will continue to be open Hardly anything of what's the internet today was promised, but who are you to decide what the internet has to become now, and that people with different ideas need to confine themselves in their own ghettos? Everyone values privacy, it's just out of social pressure if most give up so much of it. > What we're about to find out is what happens when the tide goes out and people show you what they really believe/want -- anything other than that is a form of social control, whether via browbeating or other means No idea of what you're talking about there
  - hardwaresofton 1y ago
    You were right on the second one! Facebook wasn't even a thought in my mind per say (they're not unique in that every social network wants to build a walled garden). My focus was more on the areas outside the large walled gardens -- they might be come a bunch of smaller... fenced backyards, to put it nicely.
pjc50 1y ago
> people will stop sharing knowledge for free. It doesn't have to be that way Yeah. People over-estimate the flashy threats from AI, but to me the more significant threat is killing the open exchange of knowledge and more generally the open, trusting society by flooding it with agents which are happy to press "defect" on the prisoner's dilemma. > being open in general is a miracle/luxury of functioning, economically balanced societies where people feel driven (for many possible reasons) to give back, or have time to think of more than surviving "High trust society". Something that took the West a very long time to construct through social practices, was hugely beneficial for economic growth, but is vulnerable to defectors. Think of it like a rainforest: a resource which can be burned down to increase quarterly profit.
- hardwaresofton 1y ago
  > Yeah. People over-estimate the flashy threats from AI, but to me the more significant threat is killing the open exchange of knowledge and more generally the open, trusting society by flooding it with agents which are happy to press "defect" on the prisoner's dilemma. I don't think societies are open/trusting by default -- it takes work and a lot of anti-intuitive thinking, sustained over long periods of time. > "High trust society". Something that took the West a very long time to construct through social practices, was hugely beneficial for economic growth, but is vulnerable to defectors. Think of it like a rainforest: a resource which can be burned down to increase quarterly profit. I think the trust is downstream of the safety (and importantly "economic safety", if we can call it that). Everyone trusts more when they're not feeling threatened. People "defect" from cultures that don't work for them -- people leave the culture they like and go to another one usually because of some manifestation of danger.
account42 1y ago
Fantastic work? More like contributing to the enshittification of the web.
- Analemma_ 1y ago
  I mean, LLM scrapers set fire to the commons, and when you do that, now you have a flaming hole in the ground where the commons used to be. It's not the fault of website operaters who have to act in self-defense lest their site get DDoSed out of existence.
lr4444lr 1y ago
This is inimical to the purpose of the Internet. Maybe the dream of knowledge being free and open was always doomed to fail; if knowledge has value, and people are encouraged to spend more of their time and energy to create it rather than other kinds of work, they will have to be compensated in order to do it increasingly well. It's kinda sad though, if you grew up in a world where you could actually discover stuff organically and through search.
- immibis 1y ago
  I don't think it's inevitable doom, but a realignment of incentives will probably be needed. Perhaps in the form of payment. In several EU countries it's illegal to have any internet connection without linking it to your ID card or passport in some central database, so that could also be thing - people are generally reluctant to get arrested.
xena 1y ago
Thanks! I'm gonna try and bootstrap this into a company. My product goal for the immediate future is unbranded Anubis (already implemented) with a longer term goal of being a Canadian-run Cloudflare competitor.

g-b-r 1y ago

Yes, let's turn the whole web into Facebook, what a bright future

Schiendelman 1y ago
I think we only have two choices here: 1) every webpage requires Facebook login, and then Facebook offers free hosting for the content. 2) every webpage requires some other method of login, but not locked into a single system. I read the GP comment as suggesting we push on the second option there rather than passively waiting for the first option.
- g-b-r 1y ago
  I hope you see that you shatter anonymity and the open web with that, single system or not
  - Schiendelman 1y ago
    Of course I do. But that's already gone.
    - g-b-r 1y ago
      You must be on a different internet than mine
      - Schiendelman 1y ago
        I'm not. But what are you claiming to be true?
  - hardwaresofton 1y ago
    anonymity and the open web are different things, and neither of them were promised/guaranteed to anyone on the internet. For people that value anonymity, they'll create their own spaces. People that value openness will continue to be open. What we're about to find out is what happens when the tide goes out and people show you what they really believe/want -- anything other than that is a form of social control, whether via browbeating or other means.
    - g-b-r 1y ago
      > anonymity and the open web are different things, and neither of them were promised/guaranteed to anyone on the internet. > > For people that value anonymity, they'll create their own spaces. People that value openness will continue to be open Hardly anything of what's the internet today was promised, but who are you to decide what the internet has to become now, and that people with different ideas need to confine themselves in their own ghettos? Everyone values privacy, it's just out of social pressure if most give up so much of it. > What we're about to find out is what happens when the tide goes out and people show you what they really believe/want -- anything other than that is a form of social control, whether via browbeating or other means No idea of what you're talking about there
- hardwaresofton 1y ago
  You were right on the second one! Facebook wasn't even a thought in my mind per say (they're not unique in that every social network wants to build a walled garden). My focus was more on the areas outside the large walled gardens -- they might be come a bunch of smaller... fenced backyards, to put it nicely.

pjc50 1y ago

> people will stop sharing knowledge for free. It doesn't have to be that way Yeah. People over-estimate the flashy threats from AI, but to me the more significant threat is killing the open exchange of knowledge and more generally the open, trusting society by flooding it with agents which are happy to press "defect" on the prisoner's dilemma. > being open in general is a miracle/luxury of functioning, economically balanced societies where people feel driven (for many possible reasons) to give back, or have time to think of more than surviving "High trust society". Something that took the West a very long time to construct through social practices, was hugely beneficial for economic growth, but is vulnerable to defectors. Think of it like a rainforest: a resource which can be burned down to increase quarterly profit.

hardwaresofton 1y ago
> Yeah. People over-estimate the flashy threats from AI, but to me the more significant threat is killing the open exchange of knowledge and more generally the open, trusting society by flooding it with agents which are happy to press "defect" on the prisoner's dilemma. I don't think societies are open/trusting by default -- it takes work and a lot of anti-intuitive thinking, sustained over long periods of time. > "High trust society". Something that took the West a very long time to construct through social practices, was hugely beneficial for economic growth, but is vulnerable to defectors. Think of it like a rainforest: a resource which can be burned down to increase quarterly profit. I think the trust is downstream of the safety (and importantly "economic safety", if we can call it that). Everyone trusts more when they're not feeling threatened. People "defect" from cultures that don't work for them -- people leave the culture they like and go to another one usually because of some manifestation of danger.

account42 1y ago

Fantastic work? More like contributing to the enshittification of the web.

Analemma_ 1y ago
I mean, LLM scrapers set fire to the commons, and when you do that, now you have a flaming hole in the ground where the commons used to be. It's not the fault of website operaters who have to act in self-defense lest their site get DDoSed out of existence.

lr4444lr 1y ago

This is inimical to the purpose of the Internet. Maybe the dream of knowledge being free and open was always doomed to fail; if knowledge has value, and people are encouraged to spend more of their time and energy to create it rather than other kinds of work, they will have to be compensated in order to do it increasingly well. It's kinda sad though, if you grew up in a world where you could actually discover stuff organically and through search.

immibis 1y ago
I don't think it's inevitable doom, but a realignment of incentives will probably be needed. Perhaps in the form of payment. In several EU countries it's illegal to have any internet connection without linking it to your ID card or passport in some central database, so that could also be thing - people are generally reluctant to get arrested.

xena 1y ago

Thanks! I'm gonna try and bootstrap this into a company. My product goal for the immediate future is unbranded Anubis (already implemented) with a longer term goal of being a Canadian-run Cloudflare competitor.