3 ms·

The idea of a public cache available to anyone who wishes to index it is ... kind of compelling. If it was the only indexer allowed, and it was publically gove

The idea of a public cache available to anyone who wishes to index it is ... kind of compelling. If it was the only indexer allowed, and it was publically governed, then enforcing changes to regulation would be a lot more straightforward. Imagine if indexing public social media profiles was deemed unacceptable, and within days that content disappeared from all search engines. I don't think it'll ever happen, but it's interesting to think about.

xtracto 6y ago
That would be a very cool use case for something like STORJ or IPFS.
ISL 6y ago
Imagine if Donald Trump decided that indexing Joe Biden's campaign site was unacceptable. A mandated singular public cache has potential slippery slopes.
- whimsicalism 6y ago
  Imagine if Donald Trump decided to tax campaign donations to Joe Biden's campaign at 100%. I am unconvinced by the "slippery slope" argument being deployed by default to any governmental attempt to combat tech monopolies.
  - ISL 6y ago
    This is an argument against centralization more than it is against government. "One index to rule them all" seems more fraught with difficulty than, "large cloud providers are unhappy that crawlers on the open web are crawling the open web".
    - whimsicalism 6y ago
      If the impact stopped at "large cloud providers" being unhappy, I think that you're correct. But I think we've seen considerably downstream "difficulty" for the rest of society from search essentially being consolidated into one private actor.
- passivate 6y ago
  >A mandated singular public cache has potential slippery slopes. That may be, but it seems like everything has a slippery slope - if the wrong person gets into power, or if the public look the other way/complacence/ignorance/indifference, etc, etc. It shouldn't stop us evaluating choices on their merits, and there is a lot of merit to entrusting 'core infrastructure' type entities to the government - or at-least having an option.
drivingmenuts 6y ago
> If it was the only indexer allowed, and it was publically governed Which would put it under government regulation and be forever mired in politics over what was moral, immoral, ethical or unethical and all other kerfuffle. To an extent, it’s already that way, but that would make it worse than it is currently.
tlibert 6y ago
So out law web scrapping entirely?
simantel 6y ago
Common Crawl is attempting to offer this as a non-profit: https://commoncrawl.org https://commoncrawl.org
- jackson1442 6y ago
  o/t but what the hell are they doing to scroll on that page? I move my fingers a centimeter on my trackpad and the page is already scrolled all the way to the bottom. Hijacking scroll like this is one of the biggest turnoffs a website can have for me, up there with being plastered with ads and crap. It's ok imo in the context of doing some flashy branding stuff (think Google Pixel, Tesla splashes) but contentful pages shouldn't ever do this.
  - aembleton 6y ago
    Add *##+js(aeld, scroll) to your uBO filters. That will stop scroll JS for all websites.
hackeraccount 6y ago
I'd have to look more but maybe running a cache isn't dead simple. I can imagine that the benefits of manipulating what's in the cache either adding or removing would be very high. Google and the others are private companies so they're not required to do everything in the public view. A public cache wouldn't be able - indeed shouldn't - to play cat and mouse games with potential opponents. I suspect most of the games played require not explaining exactly what you're doing.
ricardo81 6y ago
An alternative but similar idea, apply your own algorithms to a crawler/index. That's half the problem with these large platforms commanding the majority of eyeballs, you search the entire web for something and you get results back via a black box. Alternatives in general are most definitely a good thing. Knuckleheads' Club at the very least are doing a great job of raising awareness and the potential barriers to entry for alternatives.
sixdimensional 6y ago
Here's an idea... what if search became a peer-to-peer standardized protocol that is part of the stack to complement DNS? E.g. instead of using DNS as the primary entry point, you use a different protocol at that level to do "distributed search". DNS would still play a role too, but if "search" was a core protocol, the entry point for most people would be different. Similar to some of the concepts of "Linked Data", maybe - https://en.wikipedia.org/wiki/Linked_data https://en.wikipedia.org/wiki/Linked_data. The problem is getting to a standard, it would essentially need to be federated search so a standard would have to be established (de facto most likely). Also, indexes and storage, distribution of processing load.. peer-to-peer search is already a thing, but it doesn't seem to be a core function of the Internet. This is basically the same concept as making an "open" version of something that is "closed" in order to compete, I guess.
- wizzwizz4 6y ago
  https://en.wikipedia.org/wiki/YaCy https://en.wikipedia.org/wiki/YaCy

xtracto 6y ago

That would be a very cool use case for something like STORJ or IPFS.

ISL 6y ago

Imagine if Donald Trump decided that indexing Joe Biden's campaign site was unacceptable. A mandated singular public cache has potential slippery slopes.

whimsicalism 6y ago
Imagine if Donald Trump decided to tax campaign donations to Joe Biden's campaign at 100%. I am unconvinced by the "slippery slope" argument being deployed by default to any governmental attempt to combat tech monopolies.
- ISL 6y ago
  This is an argument against centralization more than it is against government. "One index to rule them all" seems more fraught with difficulty than, "large cloud providers are unhappy that crawlers on the open web are crawling the open web".
  - whimsicalism 6y ago
    If the impact stopped at "large cloud providers" being unhappy, I think that you're correct. But I think we've seen considerably downstream "difficulty" for the rest of society from search essentially being consolidated into one private actor.
passivate 6y ago
>A mandated singular public cache has potential slippery slopes. That may be, but it seems like everything has a slippery slope - if the wrong person gets into power, or if the public look the other way/complacence/ignorance/indifference, etc, etc. It shouldn't stop us evaluating choices on their merits, and there is a lot of merit to entrusting 'core infrastructure' type entities to the government - or at-least having an option.

drivingmenuts 6y ago

> If it was the only indexer allowed, and it was publically governed Which would put it under government regulation and be forever mired in politics over what was moral, immoral, ethical or unethical and all other kerfuffle. To an extent, it’s already that way, but that would make it worse than it is currently.

tlibert 6y ago

So out law web scrapping entirely?

simantel 6y ago

Common Crawl is attempting to offer this as a non-profit: https://commoncrawl.org https://commoncrawl.org

jackson1442 6y ago
o/t but what the hell are they doing to scroll on that page? I move my fingers a centimeter on my trackpad and the page is already scrolled all the way to the bottom. Hijacking scroll like this is one of the biggest turnoffs a website can have for me, up there with being plastered with ads and crap. It's ok imo in the context of doing some flashy branding stuff (think Google Pixel, Tesla splashes) but contentful pages shouldn't ever do this.
- aembleton 6y ago
  Add *##+js(aeld, scroll) to your uBO filters. That will stop scroll JS for all websites.

hackeraccount 6y ago

I'd have to look more but maybe running a cache isn't dead simple. I can imagine that the benefits of manipulating what's in the cache either adding or removing would be very high. Google and the others are private companies so they're not required to do everything in the public view. A public cache wouldn't be able - indeed shouldn't - to play cat and mouse games with potential opponents. I suspect most of the games played require not explaining exactly what you're doing.

ricardo81 6y ago

An alternative but similar idea, apply your own algorithms to a crawler/index. That's half the problem with these large platforms commanding the majority of eyeballs, you search the entire web for something and you get results back via a black box. Alternatives in general are most definitely a good thing. Knuckleheads' Club at the very least are doing a great job of raising awareness and the potential barriers to entry for alternatives.

sixdimensional 6y ago

Here's an idea... what if search became a peer-to-peer standardized protocol that is part of the stack to complement DNS? E.g. instead of using DNS as the primary entry point, you use a different protocol at that level to do "distributed search". DNS would still play a role too, but if "search" was a core protocol, the entry point for most people would be different. Similar to some of the concepts of "Linked Data", maybe - https://en.wikipedia.org/wiki/Linked_data https://en.wikipedia.org/wiki/Linked_data. The problem is getting to a standard, it would essentially need to be federated search so a standard would have to be established (de facto most likely). Also, indexes and storage, distribution of processing load.. peer-to-peer search is already a thing, but it doesn't seem to be a core function of the Internet. This is basically the same concept as making an "open" version of something that is "closed" in order to compete, I guess.

wizzwizz4 6y ago
https://en.wikipedia.org/wiki/YaCy https://en.wikipedia.org/wiki/YaCy