6 ms·

Distributed Social App with CouchDB. Brilliant

70 points by messel 16y ago

beza1e1 16y ago
There are little technical details. As far as i understand the thesis is: CouchDB provides all the groundwork for distributed state, so there is no need to build a lot sync/auth/notify code like Diaspora does. Really?
- Tichy 16y ago
  Will be interesting to see how well it works with 1 million nodes.
  - technoweenie 16y ago
    Isn't that the point with these projects? Managing a central service for 500 million users is hard. But with a decentralized setup, I won't need to connect to a million nodes. I don't know quite that many people. Course, a friend request changes from a simple button click to getting my local couchdb instance to do a filtered sync with your couchdb instance.
    - Tichy 16y ago
      You might have a point there. Although some people on Twitter have 100000 Followers and more.
      - messel 16y ago
        I think if you have 1million active followers some kind of push publishing setup could help. But as far as I know there's nothing in the tech of couchdb that prevents replicating to a million other nodes.
        maggit 16y ago
        One obstacle is that you cannot have one million open sockets (for long polling) at the same time on one node. I don't know how easy this is to mitigate.
        messel 16y ago
        Hmm. The changes are posted through comet in continuous replication? That would mean another distribution system would be better for heavily published nodes.
        wmf 16y ago
        You can have a million open sockets in Erlang, although it requires serious RAM: http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-1 http://www.metabrew.com/article/a-million-user-comet-applica...
- antileet 16y ago
  I don't think it's that capable of doing all that. As far as I understand, CouchDB does a "dumb" version-control type synchronization. Server A can copy all the data that Server B has and do it in a nice clean manner over HTTP. I can think of several disadvantages (though I may be wrong and naive): 1. If you're running a node, and I'm running a node, my data schematics might not be compatible with yours. If I write a uniform backward-compatible RPC, then it'll be a little better. 2. I can't selectively allow you to copy some data (photos) and not others. If you are a user on my database, you can access everything you have permissions to. It's hard to set fine-grained permissions. 3. Push notifications will be hard. I'd have to poll the database for changes. For that matter, the same can be achieved by putting a giant XML file on Git, and having each person set up a remote, or using a complicated set of shell scripts and rsync. Couch's replication facility is really neat, especially if you want to scale horizontally or if you're adding redundancy, etc. CouchDB is becoming the new hammer. It'll get tempting to write source control, file backup, etc, using this database because of the data sync, and decent query capabilities. But I feel that building a social network on top of CouchDB's replication might not be the best of ideas.
  - daleharvey 16y ago
    couchdb has filtered replication so you can selectively choose what gets replicated and it has continous replication with a comet changes feed, so push notifications are also built in. the data schematics are a bit of a non issue, at some point someone needs to agree with what the data format is. basically couchdb already has a lot of this functionality baked in as first class citizens, I agree that there is a lot of situations where people will be hammering couch into a hole that doesnt fit (source control would be a good one), but I certainly dont think this is one of them.
    - messel 16y ago
      Yeah I wanted to build a stand alone media sharing app (big data files, images, movies, etc), but now I may not need one. Continuous replication of parts of subscribed databases was the idea I was going to go with.
    - 16y ago
Joeboy 16y ago
There's a bit more info in the projects's README at http://github.com/maxogden/couchappspora http://github.com/maxogden/couchappspora Unlike the link submitted it suggests it's intended to be an alternative implementation of Diaspora's protocols, which for my money makes it a bit more interesting than yet another distributed social network.
- messel 16y ago
  My bad, yeah I caught that he cloned the API and updated the post this AM.
  - maxogden 16y ago
    couchappspora is shooting for ostatus/SWAT0 compatibility. I just started a separate project to work towards implementing ostatus in commonjs http://github.com/maxogden/ostatus-js http://github.com/maxogden/ostatus-js
    - messel 16y ago
      so what do you need: PuSH, Salmon, anything else? update learned to read :D OStatus consists of the following protocols: WebFinger PortableContacts Activity Streams PubSubHubBub Atom
knodi 16y ago
I love couchdb so much. It has made relocation and big data crunching so much easier.
deleted 16y ago
[deleted]
maggit 16y ago
@messel: Interesting find :) But I get a bit suspicious when the title says "Brilliant". Do you agree that removing it would be in the spirit of the submission guidelines for HN? (Though, of course, the guidelines do not say to remove such words)
- messel 16y ago
  oh brilliant was in my post title. when I came across Max's app that Guinness image was the first thing that popped up.
  - maggit 16y ago
    To clarify, I'm not suggesting that you remove the "Brilliant" from the linked post, only from the HN submission title. The Guinness reference is completely lost on me when seeing the picture, and not much clearer when just seeing the title :) I'm sure I'm not the only one...
    - messel 16y ago
      I don't mind moding the post here, but I can't seem to edit it. Admit to the odd reference, it probably doesn't click with everyone. Maybe I watch too much BBC on netflix?
mrinterweb 16y ago
Nice work Max. You should do a show and tell at tonight's pdxruby brigade.
- messel 16y ago
  Man I wished in Portland sometimes, just to go to some of your guys hack meetups.
cies 16y ago
becareful with supplying your email address! it said i had to provide it for my gravatar, but it turns out it is _publically_ visible and _unchangable_ after you entered it.. it's a trap! :-P
- messel 16y ago
  :D I don't think Max expected the implementation would get so much attention.
js4all 16y ago
This is a really good example of not reinventing the wheel. Instead built-in functions of the wonderful CouchDB are used - indeed brilliant.

70 points by messel 16y ago

There are little technical details. As far as i understand the thesis is: CouchDB provides all the groundwork for distributed state, so there is no need to build a lot sync/auth/notify code like Diaspora does. Really?

Tichy 16y ago
Will be interesting to see how well it works with 1 million nodes.
- technoweenie 16y ago
  Isn't that the point with these projects? Managing a central service for 500 million users is hard. But with a decentralized setup, I won't need to connect to a million nodes. I don't know quite that many people. Course, a friend request changes from a simple button click to getting my local couchdb instance to do a filtered sync with your couchdb instance.
  - Tichy 16y ago
    You might have a point there. Although some people on Twitter have 100000 Followers and more.
    - messel 16y ago
      I think if you have 1million active followers some kind of push publishing setup could help. But as far as I know there's nothing in the tech of couchdb that prevents replicating to a million other nodes.
      - maggit 16y ago
        One obstacle is that you cannot have one million open sockets (for long polling) at the same time on one node. I don't know how easy this is to mitigate.
        messel 16y ago
        Hmm. The changes are posted through comet in continuous replication? That would mean another distribution system would be better for heavily published nodes.
        wmf 16y ago
        You can have a million open sockets in Erlang, although it requires serious RAM: http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-1 http://www.metabrew.com/article/a-million-user-comet-applica...
antileet 16y ago
I don't think it's that capable of doing all that. As far as I understand, CouchDB does a "dumb" version-control type synchronization. Server A can copy all the data that Server B has and do it in a nice clean manner over HTTP. I can think of several disadvantages (though I may be wrong and naive): 1. If you're running a node, and I'm running a node, my data schematics might not be compatible with yours. If I write a uniform backward-compatible RPC, then it'll be a little better. 2. I can't selectively allow you to copy some data (photos) and not others. If you are a user on my database, you can access everything you have permissions to. It's hard to set fine-grained permissions. 3. Push notifications will be hard. I'd have to poll the database for changes. For that matter, the same can be achieved by putting a giant XML file on Git, and having each person set up a remote, or using a complicated set of shell scripts and rsync. Couch's replication facility is really neat, especially if you want to scale horizontally or if you're adding redundancy, etc. CouchDB is becoming the new hammer. It'll get tempting to write source control, file backup, etc, using this database because of the data sync, and decent query capabilities. But I feel that building a social network on top of CouchDB's replication might not be the best of ideas.
- daleharvey 16y ago
  couchdb has filtered replication so you can selectively choose what gets replicated and it has continous replication with a comet changes feed, so push notifications are also built in. the data schematics are a bit of a non issue, at some point someone needs to agree with what the data format is. basically couchdb already has a lot of this functionality baked in as first class citizens, I agree that there is a lot of situations where people will be hammering couch into a hole that doesnt fit (source control would be a good one), but I certainly dont think this is one of them.
  - messel 16y ago
    Yeah I wanted to build a stand alone media sharing app (big data files, images, movies, etc), but now I may not need one. Continuous replication of parts of subscribed databases was the idea I was going to go with.
  - 16y ago

Joeboy 16y ago

There's a bit more info in the projects's README at http://github.com/maxogden/couchappspora http://github.com/maxogden/couchappspora Unlike the link submitted it suggests it's intended to be an alternative implementation of Diaspora's protocols, which for my money makes it a bit more interesting than yet another distributed social network.

messel 16y ago
My bad, yeah I caught that he cloned the API and updated the post this AM.
- maxogden 16y ago
  couchappspora is shooting for ostatus/SWAT0 compatibility. I just started a separate project to work towards implementing ostatus in commonjs http://github.com/maxogden/ostatus-js http://github.com/maxogden/ostatus-js
  - messel 16y ago
    so what do you need: PuSH, Salmon, anything else? update learned to read :D OStatus consists of the following protocols: WebFinger PortableContacts Activity Streams PubSubHubBub Atom

knodi 16y ago

I love couchdb so much. It has made relocation and big data crunching so much easier.

deleted 16y ago

[deleted]

maggit 16y ago

@messel: Interesting find :) But I get a bit suspicious when the title says "Brilliant". Do you agree that removing it would be in the spirit of the submission guidelines for HN? (Though, of course, the guidelines do not say to remove such words)

messel 16y ago
oh brilliant was in my post title. when I came across Max's app that Guinness image was the first thing that popped up.
- maggit 16y ago
  To clarify, I'm not suggesting that you remove the "Brilliant" from the linked post, only from the HN submission title. The Guinness reference is completely lost on me when seeing the picture, and not much clearer when just seeing the title :) I'm sure I'm not the only one...
  - messel 16y ago
    I don't mind moding the post here, but I can't seem to edit it. Admit to the odd reference, it probably doesn't click with everyone. Maybe I watch too much BBC on netflix?

mrinterweb 16y ago

Nice work Max. You should do a show and tell at tonight's pdxruby brigade.

messel 16y ago
Man I wished in Portland sometimes, just to go to some of your guys hack meetups.

cies 16y ago

becareful with supplying your email address! it said i had to provide it for my gravatar, but it turns out it is _publically_ visible and _unchangable_ after you entered it.. it's a trap! :-P

messel 16y ago
:D I don't think Max expected the implementation would get so much attention.

js4all 16y ago

This is a really good example of not reinventing the wheel. Instead built-in functions of the wonderful CouchDB are used - indeed brilliant.