4 ms·

Achieving Scale in the Decentralized Web

66 points by pfraze 10y ago

harryf 10y ago
> There’s no walled garden for Web crawlers. Their datasets can be freely replicated from the public Web. The same can largely be said of the normal web when it comes to Google, but that doesn't alter the fact that they have a monopoly. Brand recognition, economies of scale and similar underpin their position. Can a decentralized web prevent monopoly?
- throatfuck 10y ago
  >The same can largely be said of the normal web when it comes to Google, but that doesn't alter the fact that they have a monopoly. Brand recognition, economies of scale and similar underpin their position. Well that's like saying that GNU will never succeed because Windows and Mac have such huge market shares. Obviously not true.
- pfraze 10y ago
  I guess we could ask Google to stop outperforming all their competitors. Two thoughts on this: - Maybe someday we'll find a way to do fully distributed services reliably. None of the global blockchain solutions are good enough, by my standards. What we can do is remove the friction to creating competition, which the P2P stack and other techniques (like the old Web Intents spec) are for. - I would like to see home servers become a thing. Put ten terrabytes in the corner, run a bunch of crawlers on it, include some non-public services. You can't beat google on searching the web as it exists right now, but google cant search private data.
- sharemywin 10y ago
  google bundles ranking with crawling. if I just want meta data about a set of sites that's different. Also, if your talking real-time web post-date(time) is pretty useful data for a feed.
pudo 10y ago
"If we put all economics aside, it's obvious the web should be organised this and this way..." What I'd really like to see is decentralised web proponents (of which I'm no longer one) put out some ideas for how their model is going to support economic activity of any interesting kind. Yes, selling advertising by centralising eyeballs is shit, but show me something better. The alternative is to either wait for capitalism to finally be over (spoiler alert: they're planning another season, starting January 20), or to get a second gig as a gardener to fund my web habit.
- pfraze 10y ago
  We shouldn't let unresolved economics stop us, but here's what I've written previously: https://www.reddit.com/r/Rad_Decentralization/comments/5j8782/beaker_an_experimental_p2p_browser/dbfagy0/ https://www.reddit.com/r/Rad_Decentralization/comments/5j878...
  - lewisl9029 10y ago
    > Subscription services will still be an option, even in a P2P Web, but we are trying to reduce the natural monopoly of the dataset (no walled gardens) so that might make affect service revenue by increasing competition. I'm not sure how destabilizing it will be. If a service can be destabilized by the lack of a walled garden to lock in users, then it's in society's best interest for that service to be destabilized, because that means there are other competing services that users want to use, but can't because of the size of the existing service's network effects. The status quo of services competing mainly by the sizes of their network effects is completely hostile to innovation, experimentation, and user choice. Any efforts that could disrupt this status quo has my full support!
    - pfraze 10y ago
      Completely agree
ch 10y ago
So all that's missing with the decentralized web is a centralized service to aggregate the decentralized streams? I don't see what is being addressed here other than Twitter didn't make the client be your web server (push vs pull). Sure the decentralized approach liberates the data. But without critical mass what good is that data? Maybe the example is just not compelling enough to illustrate the real point here?
- pfraze 10y ago
  The decentralized approach removes a hard binding between the app provider and the user. When you post to Twitter, you add content to their site. When you post using the P2P web, you add content to a site you own. You retain ownership of the content; the content URL remains independent of any service; and the content remains local. You can move a P2P site to a new frontend application, and you can switch to a different aggregator for a global view. For that matter, you can skip the aggregator service, and use the pure P2P arch, and even crawl locally. It's quite a bit more than "mak[ing] the client be your web server." It's about independent publishing.
  - ch 10y ago
    Sure I get that. What you call "The P2P Web", I call "The Web". Now if you want to talk about P2P Twitter, that is a different animal.
  - mark_l_watson 10y ago
    Thanks for the article and the project. I also was skeptical at first of requiring one or more centralized web crawlers to add functionality based on global knowledge of posts. Still your proposal has the advantage of user control of content. To some degree you can get this with the current web by "following" the Feeds of certain blogs and construction a timeline for yourself. What do you think of GNU Social? I am a fan and enjoy it. Off topic but, I would like to thank the people who created the Decentralized Web Conference in SF in June. Really nice conference, and I look forward to the next one.
    - 10y ago
pmlnr 10y ago
NNTP? Wasn't that kind of a solution for this?

66 points by pfraze 10y ago

> There’s no walled garden for Web crawlers. Their datasets can be freely replicated from the public Web. The same can largely be said of the normal web when it comes to Google, but that doesn't alter the fact that they have a monopoly. Brand recognition, economies of scale and similar underpin their position. Can a decentralized web prevent monopoly?

throatfuck 10y ago
>The same can largely be said of the normal web when it comes to Google, but that doesn't alter the fact that they have a monopoly. Brand recognition, economies of scale and similar underpin their position. Well that's like saying that GNU will never succeed because Windows and Mac have such huge market shares. Obviously not true.
pfraze 10y ago
I guess we could ask Google to stop outperforming all their competitors. Two thoughts on this: - Maybe someday we'll find a way to do fully distributed services reliably. None of the global blockchain solutions are good enough, by my standards. What we can do is remove the friction to creating competition, which the P2P stack and other techniques (like the old Web Intents spec) are for. - I would like to see home servers become a thing. Put ten terrabytes in the corner, run a bunch of crawlers on it, include some non-public services. You can't beat google on searching the web as it exists right now, but google cant search private data.
sharemywin 10y ago
google bundles ranking with crawling. if I just want meta data about a set of sites that's different. Also, if your talking real-time web post-date(time) is pretty useful data for a feed.

pudo 10y ago

"If we put all economics aside, it's obvious the web should be organised this and this way..." What I'd really like to see is decentralised web proponents (of which I'm no longer one) put out some ideas for how their model is going to support economic activity of any interesting kind. Yes, selling advertising by centralising eyeballs is shit, but show me something better. The alternative is to either wait for capitalism to finally be over (spoiler alert: they're planning another season, starting January 20), or to get a second gig as a gardener to fund my web habit.

pfraze 10y ago
We shouldn't let unresolved economics stop us, but here's what I've written previously: https://www.reddit.com/r/Rad_Decentralization/comments/5j8782/beaker_an_experimental_p2p_browser/dbfagy0/ https://www.reddit.com/r/Rad_Decentralization/comments/5j878...
- lewisl9029 10y ago
  > Subscription services will still be an option, even in a P2P Web, but we are trying to reduce the natural monopoly of the dataset (no walled gardens) so that might make affect service revenue by increasing competition. I'm not sure how destabilizing it will be. If a service can be destabilized by the lack of a walled garden to lock in users, then it's in society's best interest for that service to be destabilized, because that means there are other competing services that users want to use, but can't because of the size of the existing service's network effects. The status quo of services competing mainly by the sizes of their network effects is completely hostile to innovation, experimentation, and user choice. Any efforts that could disrupt this status quo has my full support!
  - pfraze 10y ago
    Completely agree

ch 10y ago

So all that's missing with the decentralized web is a centralized service to aggregate the decentralized streams? I don't see what is being addressed here other than Twitter didn't make the client be your web server (push vs pull). Sure the decentralized approach liberates the data. But without critical mass what good is that data? Maybe the example is just not compelling enough to illustrate the real point here?

pfraze 10y ago
The decentralized approach removes a hard binding between the app provider and the user. When you post to Twitter, you add content to their site. When you post using the P2P web, you add content to a site you own. You retain ownership of the content; the content URL remains independent of any service; and the content remains local. You can move a P2P site to a new frontend application, and you can switch to a different aggregator for a global view. For that matter, you can skip the aggregator service, and use the pure P2P arch, and even crawl locally. It's quite a bit more than "mak[ing] the client be your web server." It's about independent publishing.
- ch 10y ago
  Sure I get that. What you call "The P2P Web", I call "The Web". Now if you want to talk about P2P Twitter, that is a different animal.
- mark_l_watson 10y ago
  Thanks for the article and the project. I also was skeptical at first of requiring one or more centralized web crawlers to add functionality based on global knowledge of posts. Still your proposal has the advantage of user control of content. To some degree you can get this with the current web by "following" the Feeds of certain blogs and construction a timeline for yourself. What do you think of GNU Social? I am a fan and enjoy it. Off topic but, I would like to thank the people who created the Decentralized Web Conference in SF in June. Really nice conference, and I look forward to the next one.
  - 10y ago

pmlnr 10y ago

NNTP? Wasn't that kind of a solution for this?