14 ms·

Decentralising the web: Why is it so hard to achieve?

119 points by muneeb 8y ago

claydavisss 8y ago
Most users are non-technical so the entire notion is gibberish to them Otherwise, decentralization imposes performance penalties most deem unacceptable for common use. Furthermore, in the case of p2p, you can end up serving illegal content.
- tomkinson 8y ago
  This is correct and covers most current attempts at implemention. The public cannot be confused if you want new technologies to become mainstream. Selling the value of decentralization, something few ordinary people truly care enough to make behavioral or even proactive choices towards adopting, is the wrong strategy currently. That someday will change but it could be a while. The public needs other value propositions like the promise of benefit with little to no effort or mindshare to adopt. This is similar to why mobile payments faltered (and to some extent still does) in the North American market. Too many players sending out different complex messaging. No unified voice. Eventually we will eliminate the performance penalties, but mass adoption or demand for a decentralized web will require a complex multi pronged approach that includes education and extensive value adds which are easily understood.
- sametmax 8y ago
  Also decentralisation make fighting abuses hard, such as spam, flood, dos, etc.
  - dane-pgp 8y ago
    This is the correct answer. I think that what isn't immediately obvious when most people consider the big silo websites is: 1) The army of moderators that are constantly fighting to remove illegal, offensive, or commercially damaging content. (Twitter might not care about bots posting lies to sway voters, but it doesn't want a reputation that kids are following fake celebrity accounts that post beheading videos every few months). 2) The fact that these sites, by acting as middle men (and mass surveillance infrastructures), get to calculate people's reputation, which is difficult to export between sites in a meaningful way, and is hard to replicate in a decentralised/federated architecture.
  - JustMatthew 8y ago
    While I'm not a fan of Steemit, and it has a non-trivial list of flaws imho, I think they represent an interesting counter example to your claim. The various communities on Steemit all seem to focus rather passionately on protecting their hashtags and feeds from users or content that are abusive or spammy.
  - danShumway 8y ago
    On the contrary, this is one of the few areas where decentralized systems are much better than centralized ones. The problem is that moderation doesn't scale. Small communities and small instances will nearly always be better at moderation than a large platform, because they can rely on a small number of individuals to get the job done - individuals who are more likely to be consistent, more likely to pay attention to community reactions to posts, and more likely to care in general about what they're doing. It's difficult for me to think of any large platform (Facebook, Github, Twitter, Reddit, Steam, Amazon, etc...) where moderation and general content quality is better than their smaller counterparts (Mastadon, Itch.io, Gitlab, self-hosted forums, etc...). Large platforms also become large attack vectors. The same benefits of centralization for users - (content discovery, single sources of truth, and so on) make it very cost-effective and efficient to spam and harass users. With a decentralized system, spammers are less likely to care about whatever small community you're hosting. It's not impossible for them to crawl around the internet spamming everyone, but the cost of doing so is a lot higher than targeting a single platform. So centralization inevitably leads to large platforms because the market for these platforms is winner-take-all, and it almost a kind of pseudo physical law that large platforms can not be moderated well. I don't want to be absolutist about it, but I'm really having a hard time thinking of even a single exception to that rule. Maybe Wikipedia, but I'm even kind of doubtful about that - and most of Wikipedia's quality moderation comes from a group of people who are completely obsessed with the project. That mindset also doesn't scale to Twitter/Facebook sizes.
    - Kalium 8y ago
      You're right! Gitlab is much better moderated than Facebook. Small, isolated services can be policed strictly in unscalable ways. Perhaps there's another comparison to be made? Facebook's moderation, for all its many and myriad faults, is infinitely better than the stuff your average SMTP server sees every day. Federated decentralized systems are a hell that combines the worst of both centralization and decentralization. It's a great way to offer users the features they want, but at the cost of a set of systems where there's no practical way to stem abuse.
      - danShumway 8y ago
- sprucely 8y ago
  I like to read the discussion boards for MaidSafe from time to time. [1] In their case at least, serving content would not be mandatory to benefit from the p2p aspect. That's for those running a vault. And even then a vault wouldn't know what it's serving to whom. Nor would the recipient be able to determine the IPs of any of the vaults it connects to. On the performance side, I think that depends on what's being attempted and how. MaidSafers like to reiterate the notion that popular content will have chunks cached more broadly across the network, but that is only one aspect. There are other things that are not intuitive about how to develop a distributed application with no central authority. And their current model does not include sharing computing power (no Function-as-a-Service type of thing); just storage and bandwidth. [1] https://safenetforum.org/ https://safenetforum.org/
- gumby 8y ago
  > decentralization imposes performance penalties most deem unacceptable for common use... May impose such penalties; can also provide improvements (closer to point-of-use data, greater resiliency etc).
ardit33 8y ago
Napster, Limeware, Bittorrent, Kodi, Bitcoin, were/are part of the 'decentralized web'. They were build by necessity to get around copyright, monetary restrictions, censorship etc.. and not necessary for convenience. At the end of the day the average user just wants a convenient and reliable way to reach content and information. Centralization has been the best way to serve it, and I don't think it will change.
- buboard 8y ago
  Napster was by far the best way to share music, just like how popcorn time is the easiest way to get a movie right here right now. However nowadays people ditched their desktop computer/node for an iphone, and ISP upload speeds are in most places pretty bad. I think this has crippled p2p services and made centralization more convenient (while it isn't). Unfortunately we 're not gonna see decentralized web unless the general public gets back control of the compute and traffic.
  - jessaustin 8y ago
    This is probably why it's not only the Daughters Bell and Comcast encouraging FCC to stomp the brakes on modern radio tech. MPAA et al. also like to keep network consumers in a weak position. Really though it's silly for us to expect this sort of ecosystem to thrive on phones. It would more naturally thrive on the doodads that plug into TVs. That's where Kodi mainly is now, isn't it?
  - foldr 8y ago
    >However nowadays people ditched their desktop computer/node for an iphone, and ISP upload speeds are in most places pretty bad. I think this has crippled p2p services and made centralization more convenient Most people were on dialup in Napster's heyday, so I don't think that can be the issue. Music files are small.
    - buboard 8y ago
      but phones do not act as always-on nodes like napster or kazaa or torrent nodes did. battery issues + mobile data limits also
marknadal 8y ago
I spoke at the conference this week, it was mind blowing to meet all these people. :) The article seemed well done, the problem (from a information theory / network graph) though seemed weighted towards blockchain-like assumptions. This is fine, as more information needs to be seeded on this. For those interested in the title's question: Coordination might seem like the hard problem, but it is only hard because the majority of us in this community made it hard by using immutable data structures. If we use mutable data structures (you know my bias, but it is possible with https://github.com/amark/gun https://github.com/amark/gun , proof: Mitra at Internet Archive integrated it in 1 week, now decentralized IA runs on GUN), we get some other problems but they are less than O(log N) in complexity. Here is why coordination on immutability is hard: O(N^2) Just sit and think about it. Intuitively it makes sense: If I have an index to make something fast, and if it is immutable. When I "update" the index, I have to create a new index. Now the old index can't find the new index. Therefore, I need to index the index, etc. Repeat. Therefore, as long as the web never changes, a decentralized immutable web will eventually be fast, but it must be true that if while the decentralized web changes, it will be hard. This is not true if you can do decentralized mutable data. Check out our work! I hope this comment added novel insight to your day. :) If it did, let me buy you coffee next time you are in town. Cheers!
- adrianN 8y ago
  You don't have to copy the whole datastructure to update it, even if it's immutable.
  - marknadal 8y ago
    In something like an append-only log or DAG, you'd still have to index where in that log to skip to (or you'd be scanning for it every time), which is still computational complex, and not nearly as scalable as a mutable index. So yes, while you are correct, it either way is still better to use the mutable approach.
- baggins1 8y ago
  It's been updated today and Gun gets a mention!
aphextron 8y ago
The web is and always has been decentralized. But people only want to use centralized services. What does that say? Centralized services have benefits that can make them strictly superior when done ethically. We just need to come up with better ways of funding them that don’t involve building a massive survellaince apparatus. What’s easier: building a new internet, or forcing companies like Facebook to be honest and transparent?
- brokenmachine 8y ago
  > What’s easier: building a new internet, or forcing companies like Facebook to be honest and transparent? Those companies have proved they can't be trusted. So I believe the answer to that question is to build better, easy to use, decentralized platforms. I think that given a real choice, people would be happy to use open systems. Up until now they haven't been given easily usable alternatives that do all the things they want to do.
  - twblalock 8y ago
    > So I believe the answer to that question is to build better, easy to use, decentralized platforms. I think that given a real choice, people would be happy to use open systems. Up until now they haven't been given easily usable alternatives that do all the things they want to do. The problem is that decentralized systems are always going to be less usable than the centralized systems they compete with. Centralization is what makes better usability possible.
    - brokenmachine 8y ago
      > Centralization is what makes better usability possible. I disagree with this premise. There's less opportunity for monetary gain, so less effort has been invested in decentralized systems. I don't think it's an actual inherent technical limitation of decentralized systems. In fact decentralized systems promise many technical benefits. Of course there are also challenges, but in my opinion, not insurmountable ones.
twblalock 8y ago
Centralization is so useful that people will create it wherever it does not already exist. One example is Github, a centralized service that makes it easier for teams to use a source control system that was originally designed to be decentralized and distributed. When it was purchased by Microsoft and people started looking elsewhere, they looked to competing centralized solutions like GitLab and Bitbucket. Given that centralization is so useful, and people want it so much, it should not be resisted. We would be better off if the effort being put into decentralization was instead put into better centralized services that respect user privacy.
- dane-pgp 8y ago
  Monopolies are so useful that markets will create them wherever they do not already exist. One example is Microsoft, a monopolistic company that makes an operating system that everyone uses, replacing lots of unpopular incompatible ones. Given that monopolies are so useful, and people want them so much, they should not be resisted. We would be better off if the effort being put into making markets competitive was instead put into increasing Microsoft's share price.
- jessaustin 8y ago
  The problem solved by Github's centralization was establishing a namespace, which could have been done in several other ways. (In fact there are a number of other namespaces orthogonal to Github's but pointing at largely the same resources.) The problem Github solved that convinced lots of OSS people to use it was a reliable offsite that was free as in beer and less obnoxious than Sourceforge. So far it seems like some combination of VC cash and paid usage sustains Github's free service, and several other firms' free services as well. The centralization bit doesn't seem so important.
  - twblalock 8y ago
    It's a lot more than just a reliable offsite -- it invented collaboration features that didn't exist in pure Git. Github's pull request and code review workflow has become pervasive in the industry. It's also not sustained by VC cash anymore -- Microsoft is buying the company.
cameldrv 8y ago
It's already here. I'm not sure why Danielle Robinson merited a photo and a quote but there was no mention of Beaker Browser. Beaker uses Dat as its storage layer, which is similar to IPFS or Bittorrent, and then layers on data storage and sharing systems to enable services like Fritter, their Twitter clone. It's very easy to use -- just download Beaker, and you can be publishing web pages or using totally decentralized web applications.
- mattlondon 8y ago
  +1 - where IPFS and Gun present a very tech-focused entry point to the distributed web (I still do not even know how Gun works - it just says it works and that I dont need to worry etc, without much info that I could find for actually how it works e.g. how is my data shared? who runs the other nodes), Beaker presents a very friendly "on ramp" to it all that is easily understandable in a browser context. I've found it very interesting to play around with. Highly recommended.
Cypher 8y ago
Why do the articles on the site require sign up to read? prime example of friction stopping the web from being decentralized.
rvense 8y ago
Most people use products, not protocols. We need a more product-oriented approach. Decentralization and federation aren't the first or second item on the list of things most people are looking for, unless we get to "exactly like X, but decentralized." It's wrong that people don't understand or want to not depend on something like Facebook. At least where I am, a lot of people understand the issues. But there's nothing really like it, and life in my city is quite difficult without. It's nothing that can't be fixed though! I think a project like Pixelfed (https://pixelfed.org/ https://pixelfed.org/) is doing things right. We can get there.
- TeMPOraL 8y ago
  > Most people use products, not protocols. Except e-mail and HTTP. > We need a more product-oriented approach. I think this is how we landed in the current situation. Instead of protocols (IRC, XMPP), you get products (Slack, Hipchat), which are walled gardens. I wish there was a way to incentivize companies to stick to shared protocols, instead of creating their own.
  - WJW 8y ago
    The people use products, which then implement some protocol. If you go down to the nearest supermarket and ask what people use to visit their news website of choice, I'm willing to bet that 99+% will name a browser or a phone type and a very very small minority will answer HTTP, even though technically all of them use HTTP. You'll get similar answers with email and SMTP. The general public is just not very interested in the implementation details of technical products, as long as they work well enough and don't cost a lot. If you really want incentivize companies to implement open protocols, find a way for companies to make more money with the open protocol than with the walled garden, because companies is where most of the engineers are and they are the ones that will make it if the incentive is there.
  - mike_hearn 8y ago
    Moxie Marlinspike has written about this. It's hard to get people to use shared protocols because they're so hard to iterate. Companies that have a single frontend and single backend and a private protocol they control, can add new features far faster than can be done in a centralised approach. These days I'm more interested in solutions where companies produce decentralised systems, and they provide the leadership and drive necessary, with the right safeguards in place to allow their leadership to be removed and replaced if necessary. Rather than the more traditional approach of trying to not have any leadership, or do it all by committee.
hasa 8y ago
I don't see decentralization as ultimate goal. Why should it be?
- zAy0LfpBZLC8mAC 8y ago
  Because humanity's experience with centralization of power has been abysmal--and unsurprisingly so.
zaarn 8y ago
The UI/UX usually sucks a lot. Bitcoin wallets for example, suffer a lot because the average user doesn't have much of an idea that they need a strong password or what these random numbers mean other that they might behave like bank accounts. The ideal decentralized UI/UX will work largely by copy paste, with human readable names (@me@mastodon.social, for example) and doesn't have too many buttons that need more than an icon or single word to explain them. On top of that it also needs to look modern and neat while being fast and snappy on mobile devices without sucking up battery or limiting how the users can interact (ie, not possible to share photos directly with the app is a no-go) These design limitations are solved more easily by centralized services as they can just throw money at it.
rini17 8y ago
The problem here is decentralized network relies on peers to self-enforce their decent behavior. End users don't know how to, and their vendors have no skin in the game - then we get IoT botnets. ISPs won't bother with egress filtering - then we get massive DoS attacks.
nottorp 8y ago
Centralization is comfortable and in the end run, cheaper. I'd rather pay github (or depending on how much Microsoft fucks it up, a competitor in the future) a couple dollars per month than maintain my own git server. Even for services where I have to run my own server, I rent a VPS at some hosting facility rather than maintaining my own hardware or even renting bare metal. It simply doesn't make sense to run my own. And if I bothered to have a web presence, again, I'd host it somewhere. It simply doesn't make sense to maintain your own except as a hobby. Moreover, all the blockchain stuff looks to me like a solution looking for a problem instead of something generally useful. Yes, it will make sense for decentralized payments if the exchange rates ever stabilize. No, adding blockchain to everything won't help.
- kqr 8y ago
  I think key to this distinction is federation, not decentralisation. With open standards and open source implementations, you'd be able to migrate your GitHub to a different GitHub provider if Microsoft screws things up too badly. GitHub would be not a monosilo, but a set of slightly smaller siloes, all interacting seamlessly with each other.
  - nottorp 8y ago
    Well, I use github just for the repo part, no other fancy features. We run our own issue tracker on a VPS;)
- zAy0LfpBZLC8mAC 8y ago
  You are mixing up a whole lot of stuff here. The relevant question with regards to centralization is how power is distributed. In that regard, hosting your website with some hosting service is not the same thing as using Github or Facebook. Hosting your website with a hosting service has no network effect for the hosting company. If they misbehave, you can simply move to a different hosting company with absolutely no change visible to the outside. You keep your domain, you are still reachable using the same web browsers at the same URI, via the same bookmarks, via the same hyperlinks. So, while that arguably is more centralized than running your own hardware at home, it's nowhere near the level of centralization of power with services that have a business model of building a monopoly, such as Github or Facebook. If Facebook mistreats you, you cannot leave Facebook and still keep using the service they are offering. Especially with the example of git hosting, the primary problem isn't the hosting part, it's the specific business model of Github. The problem isn't that you are delegating the hosting to a specialized business. The problem is that you are delegating it to a business that's holding your hostage and using you as leverage against the general public. Also, you are making some assumptions that might be worth questioning. Like, why doesn't it make sense to maintain your own web server? I assume you assume that takes a lot of tinkering that you don't want to bother with. But why would that be the case? Is that a technical necessity? Or is that just some experience of yours that doesn't really have to be that way? Is there any reason why hosting your own server couldn't be more reliable and less work than using a hosting service?
gaius 8y ago
In the old days, you were a member of an organisation such as a university or a corporation. You had a home directory on an NFS server and in your home dir was a folder called public_html that you put your homepage in. So your URL was www.university.ac.uk/~username and your email address was username@university.ac.uk This caused problems whenever anyone moved so in the early days centralised services were seen as a solution such as Hotmail and Geocities. No one in 1995 realised how it would turn out into abusive monopolies...
cup-of-tea 8y ago
Decentralisation is hard because people crave power. As long as people crave power, people will build centralised systems of control.
- WJW 8y ago
  Also, because people crave money. It is more difficult to extract money from decentralized systems compared to centralized systems. Therefore centralized solutions often have more resources available to pay developers, marketeers and founders (and investors), which leads to increased work on the product and (hopefully) more users. It is easy to see how this would lead to a virtuous cycle where the centralized systems capture the majority of the market due to having vastly superior resources and the decentralized systems capture mostly those users and developers who care about things other than money (eg censorship resistance, etc).
  - cup-of-tea 8y ago
    Money is power. Money is quite literally the ability to make other people work for the things you want.
    - WJW 8y ago
      Money is a form of power, sure. There are different forms of power as well, not all of which can (realistically) be bought. Try purchasing serious weaponry for example (let's say a dozen tanks or fighter jets) and you will quickly discover that money is not enough.
grizzles 8y ago
It's not hard, but many in the space are working on the wrong things. We need a consensus mechanism with the integration capability of tendermint, with honey badger bft level performance. (eg: https://github.com/poanetwork/hbbft https://github.com/poanetwork/hbbft) That would be enough to build a fully distributed facebook clone and much more. It will happen, but on the timing of the engineers who know what to do to achieve it.
arendtio 8y ago
Actually, the web is decentralized. Many people mess around with the terms here. Just today I wrote a short post defining decentralized, distributed and federated in another thread [1]. While I am completely pro decentralization (I run my own Nextcloud, XMPP and E-Mail server), I am not very fond of all those 'distributed' ideas. Many of them rely on encryption which might be safe to use today, but encryption has this anoying property becoming unsafe in the future. So puting my data encrypted to some distributed network doesn't feel right for me. The problem with the tech giants is that their products are so easy to start with that many consumers just use them because they don't see real alternatives. And quite frankly, often there are no equivalent alternatives, as the tech giants throw so much money at their products that very few other products have the same ease of use. I am not sure about a solution to this dilemma, but I hope that some day we will have more high quality, easy to use, federated services. [1] https://news.ycombinator.com/item?id=17695808 https://news.ycombinator.com/item?id=17695808
- wslh 8y ago
  There are different intepretations or definitions of what is decentralized. For example, Vitalik defines the concwpts in a different way https://medium.com/@VitalikButerin/the-meaning-of-decentralization-a0c92b76a274 https://medium.com/@VitalikButerin/the-meaning-of-decentrali...
- mike_hearn 8y ago
  I've been working on decentralised systems of one form or another for a long time. I think there are several ways forward. One way is to build high quality platforms. Part of why centralised apps tend to be more common than decentralised is that we have pretty good or at least well understood platforms for making centralised apps, but the platforms for making decentralised apps are obscure at best and largely non-existent. Another way is to narrow and repoint the focus a bit. People thinking about decentralisation tend to immediately gravitate to consumer applications. I've found it easier to get traction in the business-to-business space. Despite all the noise generated by the press consumers often don't really care about things like lockin or data privacy - the tech firms protect people's data well enough, and there isn't usually enough lockin to really worry outside of maybe smartphone operating systems. However businesses are often much more sensitive to these issues. Yes they will outsource and use centralised services too, but they're much more likely to care about things like confidentiality, privacy of their data/trade secrets, they sometimes have regulatory constraints that push them in the direction of decentralisation etc. So that's one reason why I'm working on Corda, which is a platform for p2p business. It tends to get described as a blockchain system and it's got a lot of tech inspired by Bitcoin but I see it more as a platform for building decentralised apps (or more accurately, apps where you can precisely distribute and control trust points in accordance with your needs). Better platforms are absolutely the way to go for consumer apps as well. Unfortunately building such platforms will probably involve sacrificing some sacred cows. For instance the web is a terribly centralising platform. It's great at decentralising information and documents but when (ab)used as a way to build ordinary applications it creates far more centralisation than we've ever seen before. To get decentralised apps becoming commonplace we probably need to tackle the dominance of HTML5 and HTTP as a (desktop/laptop) app platform.
aethertron 8y ago
'Decentralising' means spreading around effective decision-making power. That power comes with consequences -- negatives ones, if ineptly used. And most users are quite inept, when it comes to running powerful general-purpose computers, especially internet-connected ones. I don't blame them: the internet is not a friendly environment. It's a jungle. People want to be able to safely communicate online without the responsibility of system administration. Many companies have stepped in to offer services with this trade-off. Thus centralisation has prevailed. Urbit's conceptual path to decentralisation seems sound: build a new friendly network on top of the internet, and build a new friendly server OS on top of Unix.
axilmar 8y ago
I think a decentralized web is easier than what people think. In the early days of the web, users could host their creations somewhere in their ISP's hard disks. The same model could work now, with a small difference: instead of storing only data, it may also host applications that manage those data, and collaborate with same/other applications of same/other users. For example, a photo app could allow us to manage our photos at a specific host of our choice; the same photo app uses a sharing protocol so as that the photos are shared with other users of our choice. The stored data are encrypted by our own keys and therefore not visible to third parties, we still get to share our stuff with our mates, it's decentralized because no central authority can lay claim on our data, it's protected because it is encrypted for us, but the infrastructure required for this to run is provided by a centralized entity.

119 points by muneeb 8y ago

Most users are non-technical so the entire notion is gibberish to them Otherwise, decentralization imposes performance penalties most deem unacceptable for common use. Furthermore, in the case of p2p, you can end up serving illegal content.

tomkinson 8y ago
This is correct and covers most current attempts at implemention. The public cannot be confused if you want new technologies to become mainstream. Selling the value of decentralization, something few ordinary people truly care enough to make behavioral or even proactive choices towards adopting, is the wrong strategy currently. That someday will change but it could be a while. The public needs other value propositions like the promise of benefit with little to no effort or mindshare to adopt. This is similar to why mobile payments faltered (and to some extent still does) in the North American market. Too many players sending out different complex messaging. No unified voice. Eventually we will eliminate the performance penalties, but mass adoption or demand for a decentralized web will require a complex multi pronged approach that includes education and extensive value adds which are easily understood.
sametmax 8y ago
Also decentralisation make fighting abuses hard, such as spam, flood, dos, etc.
- dane-pgp 8y ago
  This is the correct answer. I think that what isn't immediately obvious when most people consider the big silo websites is: 1) The army of moderators that are constantly fighting to remove illegal, offensive, or commercially damaging content. (Twitter might not care about bots posting lies to sway voters, but it doesn't want a reputation that kids are following fake celebrity accounts that post beheading videos every few months). 2) The fact that these sites, by acting as middle men (and mass surveillance infrastructures), get to calculate people's reputation, which is difficult to export between sites in a meaningful way, and is hard to replicate in a decentralised/federated architecture.
- JustMatthew 8y ago
  While I'm not a fan of Steemit, and it has a non-trivial list of flaws imho, I think they represent an interesting counter example to your claim. The various communities on Steemit all seem to focus rather passionately on protecting their hashtags and feeds from users or content that are abusive or spammy.
- danShumway 8y ago
  On the contrary, this is one of the few areas where decentralized systems are much better than centralized ones. The problem is that moderation doesn't scale. Small communities and small instances will nearly always be better at moderation than a large platform, because they can rely on a small number of individuals to get the job done - individuals who are more likely to be consistent, more likely to pay attention to community reactions to posts, and more likely to care in general about what they're doing. It's difficult for me to think of any large platform (Facebook, Github, Twitter, Reddit, Steam, Amazon, etc...) where moderation and general content quality is better than their smaller counterparts (Mastadon, Itch.io, Gitlab, self-hosted forums, etc...). Large platforms also become large attack vectors. The same benefits of centralization for users - (content discovery, single sources of truth, and so on) make it very cost-effective and efficient to spam and harass users. With a decentralized system, spammers are less likely to care about whatever small community you're hosting. It's not impossible for them to crawl around the internet spamming everyone, but the cost of doing so is a lot higher than targeting a single platform. So centralization inevitably leads to large platforms because the market for these platforms is winner-take-all, and it almost a kind of pseudo physical law that large platforms can not be moderated well. I don't want to be absolutist about it, but I'm really having a hard time thinking of even a single exception to that rule. Maybe Wikipedia, but I'm even kind of doubtful about that - and most of Wikipedia's quality moderation comes from a group of people who are completely obsessed with the project. That mindset also doesn't scale to Twitter/Facebook sizes.
  - Kalium 8y ago
    You're right! Gitlab is much better moderated than Facebook. Small, isolated services can be policed strictly in unscalable ways. Perhaps there's another comparison to be made? Facebook's moderation, for all its many and myriad faults, is infinitely better than the stuff your average SMTP server sees every day. Federated decentralized systems are a hell that combines the worst of both centralization and decentralization. It's a great way to offer users the features they want, but at the cost of a set of systems where there's no practical way to stem abuse.
    - danShumway 8y ago
sprucely 8y ago
I like to read the discussion boards for MaidSafe from time to time. [1] In their case at least, serving content would not be mandatory to benefit from the p2p aspect. That's for those running a vault. And even then a vault wouldn't know what it's serving to whom. Nor would the recipient be able to determine the IPs of any of the vaults it connects to. On the performance side, I think that depends on what's being attempted and how. MaidSafers like to reiterate the notion that popular content will have chunks cached more broadly across the network, but that is only one aspect. There are other things that are not intuitive about how to develop a distributed application with no central authority. And their current model does not include sharing computing power (no Function-as-a-Service type of thing); just storage and bandwidth. [1] https://safenetforum.org/ https://safenetforum.org/
gumby 8y ago
> decentralization imposes performance penalties most deem unacceptable for common use... May impose such penalties; can also provide improvements (closer to point-of-use data, greater resiliency etc).

ardit33 8y ago

Napster, Limeware, Bittorrent, Kodi, Bitcoin, were/are part of the 'decentralized web'. They were build by necessity to get around copyright, monetary restrictions, censorship etc.. and not necessary for convenience. At the end of the day the average user just wants a convenient and reliable way to reach content and information. Centralization has been the best way to serve it, and I don't think it will change.

buboard 8y ago
Napster was by far the best way to share music, just like how popcorn time is the easiest way to get a movie right here right now. However nowadays people ditched their desktop computer/node for an iphone, and ISP upload speeds are in most places pretty bad. I think this has crippled p2p services and made centralization more convenient (while it isn't). Unfortunately we 're not gonna see decentralized web unless the general public gets back control of the compute and traffic.
- jessaustin 8y ago
  This is probably why it's not only the Daughters Bell and Comcast encouraging FCC to stomp the brakes on modern radio tech. MPAA et al. also like to keep network consumers in a weak position. Really though it's silly for us to expect this sort of ecosystem to thrive on phones. It would more naturally thrive on the doodads that plug into TVs. That's where Kodi mainly is now, isn't it?
- foldr 8y ago
  >However nowadays people ditched their desktop computer/node for an iphone, and ISP upload speeds are in most places pretty bad. I think this has crippled p2p services and made centralization more convenient Most people were on dialup in Napster's heyday, so I don't think that can be the issue. Music files are small.
  - buboard 8y ago
    but phones do not act as always-on nodes like napster or kazaa or torrent nodes did. battery issues + mobile data limits also

marknadal 8y ago

I spoke at the conference this week, it was mind blowing to meet all these people. :) The article seemed well done, the problem (from a information theory / network graph) though seemed weighted towards blockchain-like assumptions. This is fine, as more information needs to be seeded on this. For those interested in the title's question: Coordination might seem like the hard problem, but it is only hard because the majority of us in this community made it hard by using immutable data structures. If we use mutable data structures (you know my bias, but it is possible with https://github.com/amark/gun https://github.com/amark/gun , proof: Mitra at Internet Archive integrated it in 1 week, now decentralized IA runs on GUN), we get some other problems but they are less than O(log N) in complexity. Here is why coordination on immutability is hard: O(N^2) Just sit and think about it. Intuitively it makes sense: If I have an index to make something fast, and if it is immutable. When I "update" the index, I have to create a new index. Now the old index can't find the new index. Therefore, I need to index the index, etc. Repeat. Therefore, as long as the web never changes, a decentralized immutable web will eventually be fast, but it must be true that if while the decentralized web changes, it will be hard. This is not true if you can do decentralized mutable data. Check out our work! I hope this comment added novel insight to your day. :) If it did, let me buy you coffee next time you are in town. Cheers!

adrianN 8y ago
You don't have to copy the whole datastructure to update it, even if it's immutable.
- marknadal 8y ago
  In something like an append-only log or DAG, you'd still have to index where in that log to skip to (or you'd be scanning for it every time), which is still computational complex, and not nearly as scalable as a mutable index. So yes, while you are correct, it either way is still better to use the mutable approach.
baggins1 8y ago
It's been updated today and Gun gets a mention!

aphextron 8y ago

The web is and always has been decentralized. But people only want to use centralized services. What does that say? Centralized services have benefits that can make them strictly superior when done ethically. We just need to come up with better ways of funding them that don’t involve building a massive survellaince apparatus. What’s easier: building a new internet, or forcing companies like Facebook to be honest and transparent?

brokenmachine 8y ago
> What’s easier: building a new internet, or forcing companies like Facebook to be honest and transparent? Those companies have proved they can't be trusted. So I believe the answer to that question is to build better, easy to use, decentralized platforms. I think that given a real choice, people would be happy to use open systems. Up until now they haven't been given easily usable alternatives that do all the things they want to do.
- twblalock 8y ago
  > So I believe the answer to that question is to build better, easy to use, decentralized platforms. I think that given a real choice, people would be happy to use open systems. Up until now they haven't been given easily usable alternatives that do all the things they want to do. The problem is that decentralized systems are always going to be less usable than the centralized systems they compete with. Centralization is what makes better usability possible.
  - brokenmachine 8y ago
    > Centralization is what makes better usability possible. I disagree with this premise. There's less opportunity for monetary gain, so less effort has been invested in decentralized systems. I don't think it's an actual inherent technical limitation of decentralized systems. In fact decentralized systems promise many technical benefits. Of course there are also challenges, but in my opinion, not insurmountable ones.

twblalock 8y ago

Centralization is so useful that people will create it wherever it does not already exist. One example is Github, a centralized service that makes it easier for teams to use a source control system that was originally designed to be decentralized and distributed. When it was purchased by Microsoft and people started looking elsewhere, they looked to competing centralized solutions like GitLab and Bitbucket. Given that centralization is so useful, and people want it so much, it should not be resisted. We would be better off if the effort being put into decentralization was instead put into better centralized services that respect user privacy.

dane-pgp 8y ago
Monopolies are so useful that markets will create them wherever they do not already exist. One example is Microsoft, a monopolistic company that makes an operating system that everyone uses, replacing lots of unpopular incompatible ones. Given that monopolies are so useful, and people want them so much, they should not be resisted. We would be better off if the effort being put into making markets competitive was instead put into increasing Microsoft's share price.
jessaustin 8y ago
The problem solved by Github's centralization was establishing a namespace, which could have been done in several other ways. (In fact there are a number of other namespaces orthogonal to Github's but pointing at largely the same resources.) The problem Github solved that convinced lots of OSS people to use it was a reliable offsite that was free as in beer and less obnoxious than Sourceforge. So far it seems like some combination of VC cash and paid usage sustains Github's free service, and several other firms' free services as well. The centralization bit doesn't seem so important.
- twblalock 8y ago
  It's a lot more than just a reliable offsite -- it invented collaboration features that didn't exist in pure Git. Github's pull request and code review workflow has become pervasive in the industry. It's also not sustained by VC cash anymore -- Microsoft is buying the company.

cameldrv 8y ago

It's already here. I'm not sure why Danielle Robinson merited a photo and a quote but there was no mention of Beaker Browser. Beaker uses Dat as its storage layer, which is similar to IPFS or Bittorrent, and then layers on data storage and sharing systems to enable services like Fritter, their Twitter clone. It's very easy to use -- just download Beaker, and you can be publishing web pages or using totally decentralized web applications.

mattlondon 8y ago
+1 - where IPFS and Gun present a very tech-focused entry point to the distributed web (I still do not even know how Gun works - it just says it works and that I dont need to worry etc, without much info that I could find for actually how it works e.g. how is my data shared? who runs the other nodes), Beaker presents a very friendly "on ramp" to it all that is easily understandable in a browser context. I've found it very interesting to play around with. Highly recommended.

Cypher 8y ago

Why do the articles on the site require sign up to read? prime example of friction stopping the web from being decentralized.

rvense 8y ago

Most people use products, not protocols. We need a more product-oriented approach. Decentralization and federation aren't the first or second item on the list of things most people are looking for, unless we get to "exactly like X, but decentralized." It's wrong that people don't understand or want to not depend on something like Facebook. At least where I am, a lot of people understand the issues. But there's nothing really like it, and life in my city is quite difficult without. It's nothing that can't be fixed though! I think a project like Pixelfed (https://pixelfed.org/ https://pixelfed.org/) is doing things right. We can get there.

TeMPOraL 8y ago
> Most people use products, not protocols. Except e-mail and HTTP. > We need a more product-oriented approach. I think this is how we landed in the current situation. Instead of protocols (IRC, XMPP), you get products (Slack, Hipchat), which are walled gardens. I wish there was a way to incentivize companies to stick to shared protocols, instead of creating their own.
- WJW 8y ago
  The people use products, which then implement some protocol. If you go down to the nearest supermarket and ask what people use to visit their news website of choice, I'm willing to bet that 99+% will name a browser or a phone type and a very very small minority will answer HTTP, even though technically all of them use HTTP. You'll get similar answers with email and SMTP. The general public is just not very interested in the implementation details of technical products, as long as they work well enough and don't cost a lot. If you really want incentivize companies to implement open protocols, find a way for companies to make more money with the open protocol than with the walled garden, because companies is where most of the engineers are and they are the ones that will make it if the incentive is there.
- mike_hearn 8y ago
  Moxie Marlinspike has written about this. It's hard to get people to use shared protocols because they're so hard to iterate. Companies that have a single frontend and single backend and a private protocol they control, can add new features far faster than can be done in a centralised approach. These days I'm more interested in solutions where companies produce decentralised systems, and they provide the leadership and drive necessary, with the right safeguards in place to allow their leadership to be removed and replaced if necessary. Rather than the more traditional approach of trying to not have any leadership, or do it all by committee.

hasa 8y ago

I don't see decentralization as ultimate goal. Why should it be?

zAy0LfpBZLC8mAC 8y ago
Because humanity's experience with centralization of power has been abysmal--and unsurprisingly so.

zaarn 8y ago

The UI/UX usually sucks a lot. Bitcoin wallets for example, suffer a lot because the average user doesn't have much of an idea that they need a strong password or what these random numbers mean other that they might behave like bank accounts. The ideal decentralized UI/UX will work largely by copy paste, with human readable names (@me@mastodon.social, for example) and doesn't have too many buttons that need more than an icon or single word to explain them. On top of that it also needs to look modern and neat while being fast and snappy on mobile devices without sucking up battery or limiting how the users can interact (ie, not possible to share photos directly with the app is a no-go) These design limitations are solved more easily by centralized services as they can just throw money at it.

rini17 8y ago

The problem here is decentralized network relies on peers to self-enforce their decent behavior. End users don't know how to, and their vendors have no skin in the game - then we get IoT botnets. ISPs won't bother with egress filtering - then we get massive DoS attacks.

nottorp 8y ago

Centralization is comfortable and in the end run, cheaper. I'd rather pay github (or depending on how much Microsoft fucks it up, a competitor in the future) a couple dollars per month than maintain my own git server. Even for services where I have to run my own server, I rent a VPS at some hosting facility rather than maintaining my own hardware or even renting bare metal. It simply doesn't make sense to run my own. And if I bothered to have a web presence, again, I'd host it somewhere. It simply doesn't make sense to maintain your own except as a hobby. Moreover, all the blockchain stuff looks to me like a solution looking for a problem instead of something generally useful. Yes, it will make sense for decentralized payments if the exchange rates ever stabilize. No, adding blockchain to everything won't help.

kqr 8y ago
I think key to this distinction is federation, not decentralisation. With open standards and open source implementations, you'd be able to migrate your GitHub to a different GitHub provider if Microsoft screws things up too badly. GitHub would be not a monosilo, but a set of slightly smaller siloes, all interacting seamlessly with each other.
- nottorp 8y ago
  Well, I use github just for the repo part, no other fancy features. We run our own issue tracker on a VPS;)
zAy0LfpBZLC8mAC 8y ago
You are mixing up a whole lot of stuff here. The relevant question with regards to centralization is how power is distributed. In that regard, hosting your website with some hosting service is not the same thing as using Github or Facebook. Hosting your website with a hosting service has no network effect for the hosting company. If they misbehave, you can simply move to a different hosting company with absolutely no change visible to the outside. You keep your domain, you are still reachable using the same web browsers at the same URI, via the same bookmarks, via the same hyperlinks. So, while that arguably is more centralized than running your own hardware at home, it's nowhere near the level of centralization of power with services that have a business model of building a monopoly, such as Github or Facebook. If Facebook mistreats you, you cannot leave Facebook and still keep using the service they are offering. Especially with the example of git hosting, the primary problem isn't the hosting part, it's the specific business model of Github. The problem isn't that you are delegating the hosting to a specialized business. The problem is that you are delegating it to a business that's holding your hostage and using you as leverage against the general public. Also, you are making some assumptions that might be worth questioning. Like, why doesn't it make sense to maintain your own web server? I assume you assume that takes a lot of tinkering that you don't want to bother with. But why would that be the case? Is that a technical necessity? Or is that just some experience of yours that doesn't really have to be that way? Is there any reason why hosting your own server couldn't be more reliable and less work than using a hosting service?

gaius 8y ago

In the old days, you were a member of an organisation such as a university or a corporation. You had a home directory on an NFS server and in your home dir was a folder called public_html that you put your homepage in. So your URL was www.university.ac.uk/~username and your email address was username@university.ac.uk This caused problems whenever anyone moved so in the early days centralised services were seen as a solution such as Hotmail and Geocities. No one in 1995 realised how it would turn out into abusive monopolies...

cup-of-tea 8y ago

Decentralisation is hard because people crave power. As long as people crave power, people will build centralised systems of control.

WJW 8y ago
Also, because people crave money. It is more difficult to extract money from decentralized systems compared to centralized systems. Therefore centralized solutions often have more resources available to pay developers, marketeers and founders (and investors), which leads to increased work on the product and (hopefully) more users. It is easy to see how this would lead to a virtuous cycle where the centralized systems capture the majority of the market due to having vastly superior resources and the decentralized systems capture mostly those users and developers who care about things other than money (eg censorship resistance, etc).
- cup-of-tea 8y ago
  Money is power. Money is quite literally the ability to make other people work for the things you want.
  - WJW 8y ago
    Money is a form of power, sure. There are different forms of power as well, not all of which can (realistically) be bought. Try purchasing serious weaponry for example (let's say a dozen tanks or fighter jets) and you will quickly discover that money is not enough.

grizzles 8y ago

It's not hard, but many in the space are working on the wrong things. We need a consensus mechanism with the integration capability of tendermint, with honey badger bft level performance. (eg: https://github.com/poanetwork/hbbft https://github.com/poanetwork/hbbft) That would be enough to build a fully distributed facebook clone and much more. It will happen, but on the timing of the engineers who know what to do to achieve it.

arendtio 8y ago

Actually, the web is decentralized. Many people mess around with the terms here. Just today I wrote a short post defining decentralized, distributed and federated in another thread [1]. While I am completely pro decentralization (I run my own Nextcloud, XMPP and E-Mail server), I am not very fond of all those 'distributed' ideas. Many of them rely on encryption which might be safe to use today, but encryption has this anoying property becoming unsafe in the future. So puting my data encrypted to some distributed network doesn't feel right for me. The problem with the tech giants is that their products are so easy to start with that many consumers just use them because they don't see real alternatives. And quite frankly, often there are no equivalent alternatives, as the tech giants throw so much money at their products that very few other products have the same ease of use. I am not sure about a solution to this dilemma, but I hope that some day we will have more high quality, easy to use, federated services. [1] https://news.ycombinator.com/item?id=17695808 https://news.ycombinator.com/item?id=17695808

wslh 8y ago
There are different intepretations or definitions of what is decentralized. For example, Vitalik defines the concwpts in a different way https://medium.com/@VitalikButerin/the-meaning-of-decentralization-a0c92b76a274 https://medium.com/@VitalikButerin/the-meaning-of-decentrali...
mike_hearn 8y ago
I've been working on decentralised systems of one form or another for a long time. I think there are several ways forward. One way is to build high quality platforms. Part of why centralised apps tend to be more common than decentralised is that we have pretty good or at least well understood platforms for making centralised apps, but the platforms for making decentralised apps are obscure at best and largely non-existent. Another way is to narrow and repoint the focus a bit. People thinking about decentralisation tend to immediately gravitate to consumer applications. I've found it easier to get traction in the business-to-business space. Despite all the noise generated by the press consumers often don't really care about things like lockin or data privacy - the tech firms protect people's data well enough, and there isn't usually enough lockin to really worry outside of maybe smartphone operating systems. However businesses are often much more sensitive to these issues. Yes they will outsource and use centralised services too, but they're much more likely to care about things like confidentiality, privacy of their data/trade secrets, they sometimes have regulatory constraints that push them in the direction of decentralisation etc. So that's one reason why I'm working on Corda, which is a platform for p2p business. It tends to get described as a blockchain system and it's got a lot of tech inspired by Bitcoin but I see it more as a platform for building decentralised apps (or more accurately, apps where you can precisely distribute and control trust points in accordance with your needs). Better platforms are absolutely the way to go for consumer apps as well. Unfortunately building such platforms will probably involve sacrificing some sacred cows. For instance the web is a terribly centralising platform. It's great at decentralising information and documents but when (ab)used as a way to build ordinary applications it creates far more centralisation than we've ever seen before. To get decentralised apps becoming commonplace we probably need to tackle the dominance of HTML5 and HTTP as a (desktop/laptop) app platform.

aethertron 8y ago

'Decentralising' means spreading around effective decision-making power. That power comes with consequences -- negatives ones, if ineptly used. And most users are quite inept, when it comes to running powerful general-purpose computers, especially internet-connected ones. I don't blame them: the internet is not a friendly environment. It's a jungle. People want to be able to safely communicate online without the responsibility of system administration. Many companies have stepped in to offer services with this trade-off. Thus centralisation has prevailed. Urbit's conceptual path to decentralisation seems sound: build a new friendly network on top of the internet, and build a new friendly server OS on top of Unix.

axilmar 8y ago

I think a decentralized web is easier than what people think. In the early days of the web, users could host their creations somewhere in their ISP's hard disks. The same model could work now, with a small difference: instead of storing only data, it may also host applications that manage those data, and collaborate with same/other applications of same/other users. For example, a photo app could allow us to manage our photos at a specific host of our choice; the same photo app uses a sharing protocol so as that the photos are shared with other users of our choice. The stored data are encrypted by our own keys and therefore not visible to third parties, we still get to share our stuff with our mates, it's decentralized because no central authority can lay claim on our data, it's protected because it is encrypted for us, but the infrastructure required for this to run is provided by a centralized entity.