4 ms·

A central-hub choice of implementation shouldn't be confused with a centralization requirement however. It is possible for distributed systems to still converg

A central-hub choice of implementation shouldn't be confused with a centralization requirement however. It is possible for distributed systems to still converge on things (indeed, entire papers have been written on the problem of distributed “agreement”), it's just that they must work in a very different way. Indeed, we had this, in the form of NNTP and older news servers; those servers essentially exchanged copies of whatever was posted from around the Internet. If there were no news.ycombinator.com, there could still be a wide variety of servers that constitute “hacker news” around the world, that come to a general agreement on what the collection of posts should be. At any instant in time, not all servers will be 100% identical but for all practical purposes they would be.

jasode 11y ago
>A central-hub choice of implementation shouldn't be confused with a centralization requirement however. >Indeed, we had this, in the form of NNTP and older news servers; You're focusing on the technical aspects. The other forces are social and economics. Both of those lead to centralization. Take your example of Usenet news groups. Yes, in a technical sense, it's decentralized but once you get economics/money involved, what happens? My ISP like Verizon doesn't carry all newsgroups (e.g. exclude newsgroups with large binaries.) Or another ISP may carry more newsgroups but the retention is 1 month instead of 1 year. So the economics converges to an central entity like Deja News that emphasizes the archival of usenet posts. Then Google buys them (Google Groups). Google Groups is now "centralized" for finding very old usenet posts. Conceivably, any of us besides DejaNews could have archived the posts but we didn't spend the money on harddrives to store it whereas DejaNews did. Hence, we end up with a centralized archive. When I had Verizon, their newsgroups feed was very poor. I therefore paid $10 extra to GigaNews because they had everything with multi-year retention. That's another "centralized" hub driven by economics. Looking only at the usenet RFC technical specs for distributing news posts doesn't reveal how the economics makes centralization emerge. As for github's centralization... A lot of us want to host a public git repository. Hmmmm... if github didn't exist, what do we do? Pay for a VPS for $9/month plan and host git on that? What about extra bandwidth costs if it's a popular repo and lots of people are cloning it? That's a lot of ugly financial bookkeeping that we don't want to do since we'd rather just concentrate on programming. Wait -- there's an ABC company out there that lets me conveniently host a git repo there? For free? Sign me up! That "ABC company" by way of economics and social dynamics has turned out to be Github and BitBucket. And since Github has been winning the mindshare, you get centralization. There is no irony in git-the-tool being distributed and github-the-focal-point being centralized.
icedchai 11y ago
Sadly, most people have forgotten about Usenet... or never knew it existed. The typical web forum is vastly inferior to most news clients.
- mbrock 11y ago
  Well, Usenet had basically no way of doing moderation, right? That's a pretty big missing feature from the perspective of forum administrators.
  - ksherlock 11y ago
    There are moderated groups. The NetNews Moderator's Handbook[1] has technical details (and more) but basically: 'Moderated' means that all postings to the newsgroup go to a mail address (e.g. comp.std.unix@uunet.uu.net) instead of appearing in the newsgroup directly. The postings are then forwarded via email to a moderator, or group of moderators, or even an automated program, who decides whether to actually inject the article into the newsgroup or to reject it as not meeting guidelines spelled out in the group's charter. 1: https://www.eyrie.org/~eagle/usefor/other/moderators-handbook https://www.eyrie.org/~eagle/usefor/other/moderators-handboo...
    - spc476 11y ago
      And there was one group, alt.hackers, that was moderated, but had no moderator, by design. It kept the signal-to-noise ratio pretty high.
    - quesera 11y ago
      Notably, though, USENET moderation was managed entirely by the addition of a single, unverified, header to the message ("Approved: cuzisayso"). This allowed for simple distribution, but equally simple circumvention. Proper moderation for anything important would require cryptographic signatures and a solid federated PKI. Maybe the world is ready, this time. :)
      - 11y ago
mbrock 11y ago
I agree that NNTP was a nicely federated system, and just a couple of days ago I was pondering the use of NNTP to publish Git pushes. Git's support for signed commits and content addressed objects is a great start for distributed, trustworthy collaboration. I still can't envision exactly how it would all function. There are lots of benefits to the GitHub model. They're a semi-trusted and well-funded entity who guarantee a highly available and fast synchronization. Sure, when it goes down everybody has a problem for a while... but with more distributed models you can get way more complicated failures, like "this object isn't seeded by a connected peer right now" or "this federated shard doesn't carry this repository" or whatever it may be. I'm really interested in the future of distributed code collaboration. I just think it's not so trivial, and so it's very understandable that people have gravitated towards a semi-centralized model that already works.
- u801e 11y ago
  > I agree that NNTP was a nicely federated system, and just a couple of days ago I was pondering the use of NNTP to publish Git pushes. Git's support for signed commits and content addressed objects is a great start for distributed, trustworthy collaboration. I still can't envision exactly how it would all function. Perhaps having something akin to a private NNTP server that requires authentication to post for each open source project, but is publicly viewable. These news servers could be configured to send IHAVE <article> to other news servers to increase visibility. There could be a newserver like gmane to aggregate the feeds from each of those private news servers. If you could get an authentication system that could work across those news servers, then you could have a decent system for keeping up with and contributing to open source projects.

jasode 11y ago

>A central-hub choice of implementation shouldn't be confused with a centralization requirement however. >Indeed, we had this, in the form of NNTP and older news servers; You're focusing on the technical aspects. The other forces are social and economics. Both of those lead to centralization. Take your example of Usenet news groups. Yes, in a technical sense, it's decentralized but once you get economics/money involved, what happens? My ISP like Verizon doesn't carry all newsgroups (e.g. exclude newsgroups with large binaries.) Or another ISP may carry more newsgroups but the retention is 1 month instead of 1 year. So the economics converges to an central entity like Deja News that emphasizes the archival of usenet posts. Then Google buys them (Google Groups). Google Groups is now "centralized" for finding very old usenet posts. Conceivably, any of us besides DejaNews could have archived the posts but we didn't spend the money on harddrives to store it whereas DejaNews did. Hence, we end up with a centralized archive. When I had Verizon, their newsgroups feed was very poor. I therefore paid $10 extra to GigaNews because they had everything with multi-year retention. That's another "centralized" hub driven by economics. Looking only at the usenet RFC technical specs for distributing news posts doesn't reveal how the economics makes centralization emerge. As for github's centralization... A lot of us want to host a public git repository. Hmmmm... if github didn't exist, what do we do? Pay for a VPS for $9/month plan and host git on that? What about extra bandwidth costs if it's a popular repo and lots of people are cloning it? That's a lot of ugly financial bookkeeping that we don't want to do since we'd rather just concentrate on programming. Wait -- there's an ABC company out there that lets me conveniently host a git repo there? For free? Sign me up! That "ABC company" by way of economics and social dynamics has turned out to be Github and BitBucket. And since Github has been winning the mindshare, you get centralization. There is no irony in git-the-tool being distributed and github-the-focal-point being centralized.

icedchai 11y ago

Sadly, most people have forgotten about Usenet... or never knew it existed. The typical web forum is vastly inferior to most news clients.

mbrock 11y ago
Well, Usenet had basically no way of doing moderation, right? That's a pretty big missing feature from the perspective of forum administrators.
- ksherlock 11y ago
  There are moderated groups. The NetNews Moderator's Handbook[1] has technical details (and more) but basically: 'Moderated' means that all postings to the newsgroup go to a mail address (e.g. comp.std.unix@uunet.uu.net) instead of appearing in the newsgroup directly. The postings are then forwarded via email to a moderator, or group of moderators, or even an automated program, who decides whether to actually inject the article into the newsgroup or to reject it as not meeting guidelines spelled out in the group's charter. 1: https://www.eyrie.org/~eagle/usefor/other/moderators-handbook https://www.eyrie.org/~eagle/usefor/other/moderators-handboo...
  - spc476 11y ago
    And there was one group, alt.hackers, that was moderated, but had no moderator, by design. It kept the signal-to-noise ratio pretty high.
  - quesera 11y ago
    Notably, though, USENET moderation was managed entirely by the addition of a single, unverified, header to the message ("Approved: cuzisayso"). This allowed for simple distribution, but equally simple circumvention. Proper moderation for anything important would require cryptographic signatures and a solid federated PKI. Maybe the world is ready, this time. :)
    - 11y ago

mbrock 11y ago

I agree that NNTP was a nicely federated system, and just a couple of days ago I was pondering the use of NNTP to publish Git pushes. Git's support for signed commits and content addressed objects is a great start for distributed, trustworthy collaboration. I still can't envision exactly how it would all function. There are lots of benefits to the GitHub model. They're a semi-trusted and well-funded entity who guarantee a highly available and fast synchronization. Sure, when it goes down everybody has a problem for a while... but with more distributed models you can get way more complicated failures, like "this object isn't seeded by a connected peer right now" or "this federated shard doesn't carry this repository" or whatever it may be. I'm really interested in the future of distributed code collaboration. I just think it's not so trivial, and so it's very understandable that people have gravitated towards a semi-centralized model that already works.

u801e 11y ago
> I agree that NNTP was a nicely federated system, and just a couple of days ago I was pondering the use of NNTP to publish Git pushes. Git's support for signed commits and content addressed objects is a great start for distributed, trustworthy collaboration. I still can't envision exactly how it would all function. Perhaps having something akin to a private NNTP server that requires authentication to post for each open source project, but is publicly viewable. These news servers could be configured to send IHAVE <article> to other news servers to increase visibility. There could be a newserver like gmane to aggregate the feeds from each of those private news servers. If you could get an authentication system that could work across those news servers, then you could have a decent system for keeping up with and contributing to open source projects.