7 ms·

Ask HN: How can I create a decentralized GNU-social-compatible website?

83 points by rayalez 10y ago

Hi! I'm working on this project, it's a reddit-like open source social network, written in django. Beta is almost complete, and now I really want to make it decentralized and compatible with GNU Social. I've never done this before, and some advice would be really useful to me at this point. Where do I start? Which technologies should I pick? Are there any great resources/tutorials about this stuff?

marknadal 10y ago
Is GNU social a spec? You might be interested in general P2P decentralized tools like https://webtorrent.io/ https://webtorrent.io/ and http://gun.js.org/ http://gun.js.org/ . There are tutorials that go with them, but nothing specific to GNU.
- Raed667 10y ago
  Thank you for the links, I'm going to try to migrate a chat app I had from WebRTC to this and see if it works.
- chriswarbo 10y ago
  > Is GNU social a spec? It follows https://en.wikipedia.org/wiki/OStatus https://en.wikipedia.org/wiki/OStatus > You might be interested in general P2P decentralized tools like https://webtorrent.io/ https://webtorrent.io/ and http://gun.js.org/ http://gun.js.org/ Would client-side solutions like these be useful for a django app? GNU Social uses federated servers (like email), so I think it would make more sense for the poster to do federation server-side too.
deleted 10y ago
[deleted]
jajern 10y ago
You'll probably want to start with the GNU Social source code and also take a look at mastodon to see how they hook into GNU Social. As for decentralizing, my thought would be that it should have been a developed around that from the start. I'm not an expert in it but I think that projects would be built around the decentralized idea and not later modified to be decentralized.
chriswarbo 10y ago
AFAIK GNU Social uses OStatus, which uses WebFinger (for identities), Salmon (for replies), ActivityStreams (for posting), etc. Some relevant links from Google: https://en.wikipedia.org/wiki/OStatus https://en.wikipedia.org/wiki/OStatus https://github.com/andstatus/andstatus/issues/378 https://github.com/andstatus/andstatus/issues/378 I'd look for some libraries which implement those specs. If you can find some in Python, then integration would mostly be a case of marshalling data between your representations and those standards. If there's no Python support, then you could either make a library yourself or use a supported language to make a standalone translation server to convert between the standard formats and some some simple raw serialisation of your data. Edit: also, security/encryption might be a hurdle. I played with Salmon years ago (using it as a plugin for the ocPortal CMS) when it was first announced; the example code was trivial to get up and running, but it required some not-so-straightforward crypto/certificate stuff to properly implement the "magic signature" part. IIRC this couldn't be done up-front by a developer; it would require some work for each deployment.
teekert 10y ago
Use IPFS [0] and take a look at Identi.ca [1]. [0] https://ipfs.io/ https://ipfs.io/ [1] https://identi.ca/ https://identi.ca/
- chriswarbo 10y ago
  http://identi.ca http://identi.ca runs on http://pump.io http://pump.io It used to run on (what is now called) GNU Social, which is what the poster is asking about.
rogerbraun 10y ago
I looked into this a while ago. Here are some notes I made: - https://git.heldscal.la/roger/the_wired/wiki/OStatus https://git.heldscal.la/roger/the_wired/wiki/OStatus - https://git.heldscal.la/roger/the_wired/wiki/Salmon https://git.heldscal.la/roger/the_wired/wiki/Salmon Hope it helps. Essentially, you can basic functionality going by implementing webfinger and an atom feed. This old guide might be helpful: https://web.archive.org/web/20120306210855/http://ostatus.org/2010/10/04/how-ostatus-enable-your-application https://web.archive.org/web/20120306210855/http://ostatus.or...
chriswarbo 10y ago
Not directly answering your question, but does anybody know of any server software I can run to take part in such decentralised/federated/etc. networks, which isn't complicated by multi-user, Web-frontend bloat? GNU Social, pump.io, etc. and even Matrix, XMPP and IRC, etc. all seem to assume I want thousands of simultaneous users. I don't; I'm the only user, but I want to integrate into the network. Signing up for an account on someone else's server seems to defeat the point, yet I haven't found anything which I can run persistently on my server, access in a simple way (e.g. via SSH, a shared secret, a named pipe, etc.), and doesn't make me uncomfortable allowing it online (i.e. dynamic Web stuff). I used to use Laconica (AKA StatusNet AKA GNU Social) a lot via XMPP, but then identi.ca switched off their bridge, and then switched to pump.io which doesn't seem to work with anything other than a Web frontend. How would Usenet compare? I never got into it, but hear it's filled with spam :(
- rakoo 10y ago
  You're right, most systems assume multiple people want to register to the same instance of a server; I know twister (http://twister.net.co/ http://twister.net.co/) is not like that and is expected to be run as a single user
  - chriswarbo 10y ago
    Very interesting. Going completely P2P avoids making a client/server distinction, and I could still run it continously on a server just in case connecting sporadically would miss content. I don't like the use of a blockchain though; it's a heavy requirement for something as simple as nickname assignment, which would actually be more flexible if it were done in the client anyway (in the same way that my Pidgin buddy list assigns names of my choosing to XMPP IDs).
- mxuribe 10y ago
  If I'm not mistaken, with Matrix you do not have to run a front-end per se. I believe you can simply setup a matrix home server, and then interact via an open source mobile/desktop/web application. This is not unlike old school email, where you can host your own email server, but manage email via any mobile/web/desktop email clients (of which you don't need to host the client side) . For example, I'm running a private (matrix) synapse home server for ONLY my family; as a test. I only interact via the mobile (open source) riot application (formerly named vector.im), though they do also offer a web application at https://riot.im https://riot.im . I simply point the client to my server's (appropriate matrix) url, and am off to the races - again internally for my family and I only at this point. My plan eventually is to keep running my own matrix home server, and open it up so that my family and I keep using any old client (like https://riot.im https://riot.im or whatever) BUT begin to interact with other users using matrix clients/servers, etc. My understanding is that what you wish to do ("...I'm the only user, but I want to integrate into the network") is actually quite common for folks beginning to use matrix. Give this a look-see: http://matrix.org/docs/projects/try-matrix-now.html http://matrix.org/docs/projects/try-matrix-now.html Alternatively - less ideal of course but - you can always lurk around on certain matrix-related rooms on the first/original matrix.org home server, and ask questions (on how you can set up your own server, etc.). Because I'm still in my own test phase, that's what I'm doing. Here are a few channels (via riot.im): * https://riot.im/app/#/room/#matrix:matrix.org https://riot.im/app/#/room/#matrix:matrix.org * https://riot.im/app/#/room/#riot:matrix.org https://riot.im/app/#/room/#riot:matrix.org * https://riot.im/app/#/room/#homeowners:matrix.org https://riot.im/app/#/room/#homeowners:matrix.org Good luck!
paroneayea 10y ago
So I have quite a bit of bias, since I work on a specification in this area. So yes, if you want to make sure it's GNU Social (formerly StatusNet) compatible specifically, use OStatus, which is a number of protocols combined in a specific way. The most unfortunate part of all this though is that all these different applications, GNU Social, Diaspora, Pump.io, etc all speak different protocols and aren't all really interoperable. This leads to a "fractured federation", which really isn't great. I've been participating in the W3C Social Working Group to try to standardize federation. Here's our spec, ActivityPub: https://www.w3.org/TR/activitypub/ https://www.w3.org/TR/activitypub/ Some background: it's derivative of the "Pump API", which is sort of the successor of OStatus, in the sense that the primary author of StatusNet (now GNU Social) wrote the Pump API in response to some perceived shortcomings in OStatus. We've done a lot of analysis to make sure that the protocol can handle all the kinds of things that you expect from the "big player" social networks, but in a federated setting. Interested? Now's a great time to research and give feedback. We're looking to move to Candidate Recommendation status shortly, which means we're looking for feedback and implementations. PS: I'm also involved in MediaGoblin, and we definitely intend to implement there. If you're interested in a bit of news from that end, here's a moderately fresh blogpost, including info on how to give your feedback: http://mediagoblin.org/news/tpac-2016-and-review-activitypub.html http://mediagoblin.org/news/tpac-2016-and-review-activitypub... We're planning on using it for MediaGoblin.
- staz 10y ago
  https://xkcd.com/927/ https://xkcd.com/927/
  - unimpressive 10y ago
    This comic may have singlehandedly done terrible damage to technology standards, by spreading the meme that trying to create a unified standard never works and nobody should try.
    - kwhitefoot 10y ago
      Anyone put off by a comic is clearly not suited to the hard slog that is required to get standards written, agreed, and implemented. So i think xkcd may be doing us all a favour by discouraging those who haven't got the stamina.
clacke2 10y ago
You're on Python, but it might make sense to look at the choices made by https://github.com/hotsh/rstat.us https://github.com/hotsh/rstat.us and https://github.com/gargron/mastodon https://github.com/gargron/mastodon , to get some ideas. They're in Ruby, but the language is more similar to Python than PHP is, and some of the tooling and libraries in Python and Ruby are inspired by each other too. It always helps to look at those who came before you.
mxuribe 10y ago
If you are interested in continuing to leverage your python skills (I assume this because of your ref. to Django), then perhaps building a bridge between Matrix and Gnu Social would be something BOTH communities would benefit from: http://matrix.org/docs/projects/try-matrix-now.html#servers http://matrix.org/docs/projects/try-matrix-now.html#servers
cabalamat 10y ago
I wonder whether it would make sense for work in this area to (like Git) distinguish between plumbing and porcelain. So once the plumbing is written (with a reference implementation in Python or similar) it would hopefully be an easy matter for people to write porcelains to top of it.
dethos 10y ago
I've been thinking about building some reusable django app that could be used in other projects to integrate and make them compatible with gnu social, but I never put the idea into practice (I still think it is a good idea). Recently someone posted here on HN a compatible server (https://github.com/Gargron/mastodon https://github.com/Gargron/mastodon), it isn't what you are searching for, but these 2 posts about the topic in a short period of time shows there is an certain interest. Maybe its the push I need to start building that reusable app. Regarding your question, you should start by looking at https://www.w3.org/community/ostatus/ https://www.w3.org/community/ostatus/ and the wiki.
dorfuss 10y ago
Have a look at Diaspora, it could be at least an inspiration: https://en.wikipedia.org/wiki/Diaspora_(social_network) https://en.wikipedia.org/wiki/Diaspora_(social_network)

chriswarbo 10y ago
> If I'm not mistaken, with Matrix you do not have to run a front-end per se. I believe you can simply setup a matrix home server, and then interact via an open source mobile/desktop/web application. If by "front-end" you mean Web server, then as far as I'm aware none of these networks (GNU Social, pump.io, Matrix, XMPP, email, etc.) require one per se. The problem is finding an implementation which doesn't bundle one. Looking at the servers listed on matrix.org, it appears that Pallium makes this separation: you can run Pallium stand-alone, and connect to it using client applications (which may or may not be Web-based). Unfortunately, browsing the code for all of those servers, they all seem to include facilities for user registration, authentication, etc. Maybe that's a requirement of the way Matrix works? I know that, say, an IRC server which only allows a single user would be a lonely place, since chatrooms are hosted by a particular server which all participants must connect to (logically, at least; I know there are ways to implement redundancy like FreeNode does).
- mxuribe 10y ago
  Head on over to: https://github.com/matrix-org/synapse#synapse-installation https://github.com/matrix-org/synapse#synapse-installation ...Then scroll to the last paragraph of this section, "...By default, registration of new users is disabled." The little command line method is what i used to manually register the (3) accounts for my family on my private matrix server. If you plan on using it only for yourself, it might be best to keep registration disabled, and only add yourself manually via the command line...and then you would be done. As far as "web server", I think i know what you mean now, and admittedly I have my private matrix home server running behind nginx...but that's only because I've set it up via DigitalOcean, and simply picked one of their pre-set "LEMP" server images. I believe others have installed/run a matrix home server without nginx or apache, etc. Again, since this is all just a test for my family, i literally went into my DigitalOcean account, generated a new droplet (what they call their virtual servers), and picked a "LEMP" pre-set server. From there, i went installing the synapse server. Because nginx was already installed, I did have to forward/proxy stuff from nginx to synapse...I fully expect to kill this test matrix server off. Now having lived this, I would not have picked the LEMP image, instead just use a basic linux server from DigitalOcean and then install synapse (and any associated requirements/dependencies), that way it doesn't have nginx/apache, etc. I have no experience with Pallium, but I suppose because there are several different implementations (python-based, Go-based, etc.) of the matrix homeserver, there could be different nuances for implementation. Side note: The DigitalOcean droplet that I'm using for my private test matrix home server is the $5/month version (w/512MB memory, 20GB storage). Once I create the production matrix homeserver for my family, I was thinking on using the next level up $10/month version (1GB memory, 30GB storage).
  - chriswarbo 10y ago
    > "...By default, registration of new users is disabled." It's good to know that it can be disabled, but I'm mostly concerned with attack surface, minimalism/hackability and not getting caught on an upgrade treadmill. At the moment my personal Web site is static, the Web UI to my git repos is static and read-only, pushing to those repos is done via SSH, I accept patches by email, etc. I'm hesistant to run anything dynamic at all, let alone something more complex than I need (e.g. a Web frontend) with the capability to perform destructive actions which I don't need (even if they're marked 'disabled'). It would be nice to have an Internet-facing server which only deals with untrusted data, accumulates incoming updates to a file (e.g. sqlite), pushes out posts from a queue (e.g. a directory) and serves the necessary public info when requested, maybe using static files for stuff like my profile. I can then read and write those files over SSH, maybe transforming them into some easy to consume format like maildir. Of course, the first rule of security is don't write it yourself, but this is beginning to sound like a nice weekend project :)
  - chriswarbo 10y ago
    > As far as "web server", I think i know what you mean now, and admittedly I have my private matrix home server running behind nginx Sorry, I should have clarified: when talking about "web" stuff, I meant hyperlinked HTML, intended for human consumption. Whilst sending/receiving HTTP is how most of these protocols work, at least parsing/printing JSON data in a standard format only presents quite a small attack surface, and can be set up "once and for all" (maybe tweaked as the standards slowly improve). It's quite another thing to be serving HTML, since it (like everything user-facing) tends towards bloat and churn (everyone has different tastes). That's fine, but it shouldn't require a server upgrade to fiddle with the UI (or complicating the server with dozens of configuration knobs that have nothing to do with its operation). In contrast, config files on disk are a safer way to perform admin jobs than accepting POSTs from the Internet, and using a separate client (dedicated, multi-protocol, bridging to another format, native or Web, etc.) means that the UI bloat and churn doesn't affect the server.