5 ms·

Gun – Distributed, embedded graph engine

108 points by marknadal 13y ago

beshrkayali 13y ago
Regardless of the title being confusing or not, this seems pretty interesting! Would like to see how it performs under some heavy load / multiple sources of data.
_dark_matter_ 13y ago
This title is bad. It's not an embedded database engine. It's not a database at all. The author mentions this directly in the article (How are you different from every other database that is trying to reinvent the wheel? 1. Because gun is not a database (NoDB), it is a persisted distributed cache.) There are other possible issues, but this short summary doesn't give nearly enough information about the system to make a possible determination on its efficacy. I will be interested to see further work.
- marknadal 13y ago
  Hey _dark_matter_, Thanks for the comment! I'll be doing follow up posts soon on the technical details of my conflict resolution algorithms, and will be preparing Jepsen tests for gun. I just wanted to formally announce this now, though, as I have a tendency to overly-perfect stuff, and getting feedback early on is great. Mind listing things you would like me write in-depth posts on? Thanks for the feedback!
  - _dark_matter_ 13y ago
    Hey Mark, I'm interested in several different features. First off, you make a great point about "Hosting your own database is a pain" and/or expensive. I'm sure you know about the raging debate that has been happening between MapReduce and Parallel DBs, and a big part of the reason that MapReduce (Hadoop) style has had so much support is because it's easy and cheap. In lots of instances it may run slower, but why pay for the hassle when you can just run Hadoop for free (not including server costs)? You're making the same point here. Whats the threshold on money savings? How much does it cost to use a DaaS (SQL or NoSQL) compared to what you want to do with Gun/Redis? (Also, is this going to be open source?) Next, the conflict resolution should be interesting. What kind of eventual consistency guarantees are you hoping to have? I see you're planning on addressing this. I'm also interested in the querying. That is, are you using Javascript? My problem with what you're saying about current query languages is that it's not really that hard to get the bare usage out of them. Simple queries in SQL are extremely straightforward. In addition, declarative languages really make things easier for the programmer, not harder. Who wants to have to deal with all the nuances of a join? Certainly not me.
    - marknadal 13y ago
      Realistically I won't have cost-comparisons for a while, as this project is just in the infancy and obviously needs to stabilize/mature first. Although as a quick summary, your costs should approximately be server(s) that are expensive enough to fit your smallest set of per-user active-data, plus your S3 storage amount, plus S3 API calls (I've made this part a simple options parameter, so you control how often these calls are being made - the more frequent, the more integrity from worst-case disruptions but also more expensive, less frequent and your costs go down). What is worst-case disruption? Everything goes offline simultaneously. User's don't have localstorage fallbacks in their phone/browser so retries rely on the server cache. The server cache is running the default method, not Redis - and then your server crashes or the machine goes under. And finally, of course, there is an S3 outage and you only persisted to 1 region, not multiple regions. I'll get into hairy situations like that in some of my following posts, but too much for here. Yes, you have another good point about query languages. I agree with you very much - somebody else commented about this, check out my reply to muaddirac. Please keep in touch with more questions/comments, or even email me!
- opendais 13y ago
  Exactly. "...this isn't a database" :/ Distributed caches aren't new either. [e.g. https://github.com/golang/groupcache https://github.com/golang/groupcache ] Can we get a title change to something that labels it a Distributed Cache? [e.g. Shown HN: gun – Massively Distributed Cache ]
  - marknadal 13y ago
    opendais, I'm curious if these other distributed caches actually support persistence? I wasn't able to find such a system. But I'm limited in my research, I'd love to learn more about other people's work! Please share more!
    - opendais 13y ago
      http://ehcache.org/ http://ehcache.org/ http://ehcache.org/documentation/configuration/fast-restart#strategy-options http://ehcache.org/documentation/configuration/fast-restart#... Based on a quick googling, this looks like it would be an example of that. Frankly, I don't normally want my cache to persist so it isn't something I know much about. [e.g. I store persistent data in Cassandra or MySQL]
      - marknadal 13y ago
        You hit on a really good point, in the several years that I've worked in the synchronization field (I do consulting) I've found that it has actually been harder to deal with /what you don't want to sync/ than with what you want to sync. It would be really bad if our temporary variables that are used on the client and server to do transformations on our actual data got somehow jumbled up into our real data. I have some clean delineations for this, but wow - I didn't know that this was going to explode so quickly. If I had known, I would have prepared some more resources and materials to explain this stuff. That said, everybody who signs up on the mailing list will get notified about the posts I will write on this! I'm looking forward to getting some serious counter-arguments/refutations to challenge gun.
        13y ago
noelwelsh 13y ago
Nice idea which mirrors a lot of my current thinking; far more details are needed. For example: "All conflict resolution happens locally in each peer using a deterministic algorithm." Hrrrmm.... if the model is to use CRDTs for conflict free resolution I can believe it; if it's timestamps, for instance, I'm much more doubtful.
- marknadal 13y ago
  noelwelsh, thanks for the comment! Timestamps do fail pretty miserably for handling conflict resolution - while my algorithm does use timestamps, the timestamp is hardly responsible for the actual synchronization. It just provides a basis to initially sort things off of, and then the conflict resolution kicks in. Obviously I'm going to need to do a pretty detailed post on how this works, cause unless it can be battled-tested and skeptically investigated by others, it is only as good as snake oil. So more on this soon! Thanks for bringing it up.
  - avodonosov 13y ago
    Conflict resolution is the main question.
adambard 13y ago
This actually sounds like a pretty cool and useful project, but I had to force myself to overcome the site, which I think does a poor job of marketing. The headline is a bit confusing -- it's not obvious that something can be "embedded" and "distributed", let alone "massively." You might rewrite it to tell me what the project does, not what it is. Besides that, as a developer, what I want to know is: - Why should I use it? (Instant in-browser storage synced to a distributed server network) - How is it implemented? (it seems to be a javascript project, but that's only mentioned once) - How do I run it? (It run daemons on servers and stores the data in S3) - What do I have to do to configure it? (Seems to me like "The granularity and frequency of these snapshots can be tweaked by you" means "Prepare to get elbow-deep into telling gun how to shard") If you just take the answers to these questions and make them either bullet points up above the video, or headers for your marketing essay, it would help people like me digest the project enough to be interested and read more. Finally, while I'm being a grinch, I'd adjust your styles just slightly to make the site more readable. At the very least: - Make `#main` narrower; perhaps 700px or even less (long lines are hard to read) - In `body`, apply a line-height of maybe 20pt (the lines seem too close together for the font size) - Also in `body`, drop the text-shadow. Shadow on white tends to just make things look blurry. All that said, I am interested in learning more. A self-hosted Firebase would be a real boon for many projects.
- marknadal 13y ago
  Great suggestions! Already fixed one of the things, and working on updating the others! (Although content will take a while longer). Thank you so much.
reillyse 13y ago
I'm withholding my judgment on the tech - it all seems a little too good to be true - but the copy on your page is great, I actually read every line of it which rarely ever happens when I visit a technical page.
jobeirne 13y ago
I found the header image a bit disconcerting, and immediately hit Back (I'm at work).
- marknadal 13y ago
  sorry :/ I'm trying to intentionally convey that this tool is dangerously powerful. I don't mean to offend anyone though.
  - deleted 13y ago
    [deleted]
  - jauer 13y ago
    And here I thought you were showing your good taste in customized 1911 variants... You might want to consider a name that reflects the distributed nature over the dangerous power because of reactions like the parent (although I strongly disagree with the sentiment expressed). Seriously though, this sounds cool and I'm looking forward to trying it out once you release code.
    - marknadal 13y ago
      :P I'm coming to admire guns a lot more because of this library - respect!
  - userbinator 13y ago
    I'm not in the "guns are offensive" line of thought but I think it could have a better name than an existing noun. (Although CloudFlare has already taken "railgun"...)
- chadillac 13y ago
  Pictures of guns are offensive and dangerous at work now? Really?
  - TallGuyShort 13y ago
    Heck, I used to work in an office where most people had similar weapons on them at all times.
johne20 13y ago
How would gun handle user auth and data that can only be seen by the authenticated user?
- marknadal 13y ago
  So I'm making sure that the concepts of piping and transforms are built right into the API, so you'll be able to easily filter out data by whitelists or blacklists as it is getting pushed down to your users. I'll also be writing plugins to gun that will allow people who like ORMs to just attach one on, and the filtering and validation will be handled for them. Does this answer your question?
  - johne20 13y ago
    Yes conceptually it does, I would be interested to see how it is implemented. Thanks!
muaddirac 13y ago
Looks interesting! > you never have to learn some silly separate query language again. A query language which just attempts to be some DSL to RPC another machine into doing the same query you could have already written in half the time it took to learn the query language. Not sure I agree with this sentiment - most programming languages aren't declarative like query languages are, and that seems especially useful for, well, querying.
- marknadal 13y ago
  This is true. The neat thing about the modular design I have for gun is that people can always write a plugin that receives some/any query language, and then translates it directly into the appropriate algorithms. So you should be able to write your own abstractions ontop - but this is only possible because you are able to write the direct queries underneath.
mallyvai 13y ago
"Because face it, any sufficiently capable query language has to be Turing complete" - No, no it doesn't. Stock SQL is not Turing complete. There are extensions that support recursion to make it so, but it is perfectly possible and capable of doing everything you could reasonably want without it. Caching is hard. Consistency is hard. Peer to peer is hard. I hope the author is addressing these in a sane, verifiable way. I'm really curious to see the demo apps that come out of this.
- marknadal 13y ago
  I apologize, I wasn't saying every query language /is/ Turing complete, just that to do exponentially more complicated tasks in a single atomic query... the lines start to blur between the two. Yes, these are hard subjects - especially things like cache invalidation. You raise good points, and I hope to answer them in the follow up posts I write - I expect your's and other's good eye to check my work. I'll let people know when I've written them. Demos are coming too!
jellicle 13y ago
The name "gun": a) has negative connotations, b) is ungoogleable, c) can't even write about it without being ungrammatical and d) will be filtered out at some workplaces. Dumb, de-dumb dumb dumb.
stewars 13y ago
What license is the code distributed under? It is not clear from the site or the code that is available through npm.
curveship 13y ago
So CAP: which does gun throw off the island? Based on this statement ... > It bridges the distance with a realtime connection, so updates propagate at the speed of the raw pipes linking them. ... I'm assuming it's the P.
- dantiberian 13y ago
  CA system's don't exist so it must be CP or AP. However, further consideration shows that CA is not really a coherent option because a system that is not Partition- tolerant will, by definition, be forced to give up Consistency or Availability during a partition. https://foundationdb.com/white-papers/the-cap-theorem https://foundationdb.com/white-papers/the-cap-theorem
  - GregorStocks 13y ago
    It could also be neither, which is a popular choice these days.
- marknadal 13y ago
  Out of the box gun only guarantees you eventual consistency, not strong consistency. The reason why is because the majority of web applications out there today are manipulating data sets that aren't particularly significant or regulated (tweets compared to medical records, profile updates compared to payment information). As a result, gun opts for high availability so response times are fast for users, and assuring their data won't be lost. However, the strength of this design is it allows you to implement stronger forms of consistency on top - which is necessary for those occasional pieces of data (payments, medical records, etc). But if you do this then the availability of that data diminishes because servers must wait for consistency to be achieved before doing any further transformations. The sweet spot about this is that it will only be the availability of that particular property (or properties that it depends upon) that decreases, all other pieces of information will still be snappy fast because gun is capable of locking at a granular level. This will not be as "easy" as it is in databases that are strongly consistent to begin with, but those databases aren't capable of being eventually consistent either (because that would violate the DB's principles). But it is possible, and you the developer get the control and flexibility over that. So you win some and you lose some, but I believe this setup is highly optimized because generally speaking the things that you need to be "strongly consistent" anyways have much slower "weakest links" than the speed of networks being up or down - such as approval by some human overseer (a loan officer, a lawyer, a doctor, etc.), and meatspace brains take much longer to process regulatory issues than the potentially couple of hours the network might be down. I'll touch more on these quirks in a separate post after I address conflict resolution.
xkarga00 13y ago
Not everyone is familiar with the concept of NoDB so this paper [1] is a good read to start with. [1] http://bit.ly/RjBl9S http://bit.ly/RjBl9S
Allower 13y ago
I have been looking for something just like this! Fantastic
ilaksh 13y ago
This reminds me a little of Meteor and a bit more of ShareJS. And it reminds me of a few other things. As far as transformations and conflict resolution, I am wondering, are you using something like operational transformations?
- marknadal 13y ago
  I had to develop my own synchronization algorithms after doing a lot of research on OT for a couple of reasons. One is that Google sometimes relies on being a centralized authority (Google's server) to resolve some conflicts, especially for collaborative rich text editing (Neil Fraser, a sync genius that Google hired, has a great talk on his own type of implementation "diff-match-patch", although Google uses different algorithms now I believe). Another is that OT sometimes requires you writing your own transformation commands, and I wanted something that was generally applicable to most data types without forcing developers to write their own. (It has been a while since I did a lot of this reading, so this comment may not necessarily be accurate/true, please somebody correct me!) Instead, I developed my own method, called "Analytical Fluctuation" - but I haven't written papers on it yet, but need to soon! One requirement is that it has to be truly peer to peer and cannot rely (even occasionally) on some centralized server or leader election setup. Another requirement is that it has to be latent proof, which for me means "preparing for the future" when people need to collaborate on documents not only from the other side of the world but also from Mars. This is where Neil Fraser says his system breaks down, because if there is too much latency, the patches stop applying idempotently. Again, more on this in upcoming posts.
asimpletune 13y ago
Hey, I'm confused by the statement, "No amount of leader election and consensus algorithms can patch this without facing an unjustified amount of complexity. Gun resolves all this by biting the bullet - it solves the hard problems first, not last. It gets data synchronization and conflict resolution right from the beginning, so it never has to rely on vulnerable leader election or consensus locking". My question is, how do you solve data synchronization and conflict resolution, without using the techniques that do that, i.e. leader election, or some other type of consensus algorithm?
- marknadal 13y ago
  Gun's conflict resolution algorithm is deterministic, meaning that it will choose the same answer on every peer without having to communicate with others which value it chose. This even works when the ordering of incoming updates is switched depending upon which server was the "I" sending updates out to "you" (pronouns, analogously, are inverse of each other depending upon which person is doing the talking). Uh, this is not clear/sounds confusing, I'll explain it better in my post that goes over how the resolution algorithm works. (Basically the gist is that sometimes for two servers to agree on the "same" answer they both have to have an algorithm which results in an inverse condition from what the other would answer - similar to how quantum-entangled particles have opposite spins of each other to "balance out") Point being, this makes gun truly peer to peer, because it behaves correctly if its the only one running, or if there are numerous guns interconnected with each other. No leader election, no consensus algorithm - you don't need those, because the system agrees as soon as the update are received because it resolves them immediately with an idempotent algorithm. Make sense? More details on this soon.
  - asimpletune 13y ago
    Then I look forward to your next post. Can you give me a preview on what type of failure model Gun can support? Consensus is easy assuming there are no faults, but if faults are possible, then leadership election, paxos, whatever, something is needed.
    - 13y ago

108 points by marknadal 13y ago

Regardless of the title being confusing or not, this seems pretty interesting! Would like to see how it performs under some heavy load / multiple sources of data.

_dark_matter_ 13y ago

This title is bad. It's not an embedded database engine. It's not a database at all. The author mentions this directly in the article (How are you different from every other database that is trying to reinvent the wheel? 1. Because gun is not a database (NoDB), it is a persisted distributed cache.) There are other possible issues, but this short summary doesn't give nearly enough information about the system to make a possible determination on its efficacy. I will be interested to see further work.

marknadal 13y ago
Hey _dark_matter_, Thanks for the comment! I'll be doing follow up posts soon on the technical details of my conflict resolution algorithms, and will be preparing Jepsen tests for gun. I just wanted to formally announce this now, though, as I have a tendency to overly-perfect stuff, and getting feedback early on is great. Mind listing things you would like me write in-depth posts on? Thanks for the feedback!
- _dark_matter_ 13y ago
  Hey Mark, I'm interested in several different features. First off, you make a great point about "Hosting your own database is a pain" and/or expensive. I'm sure you know about the raging debate that has been happening between MapReduce and Parallel DBs, and a big part of the reason that MapReduce (Hadoop) style has had so much support is because it's easy and cheap. In lots of instances it may run slower, but why pay for the hassle when you can just run Hadoop for free (not including server costs)? You're making the same point here. Whats the threshold on money savings? How much does it cost to use a DaaS (SQL or NoSQL) compared to what you want to do with Gun/Redis? (Also, is this going to be open source?) Next, the conflict resolution should be interesting. What kind of eventual consistency guarantees are you hoping to have? I see you're planning on addressing this. I'm also interested in the querying. That is, are you using Javascript? My problem with what you're saying about current query languages is that it's not really that hard to get the bare usage out of them. Simple queries in SQL are extremely straightforward. In addition, declarative languages really make things easier for the programmer, not harder. Who wants to have to deal with all the nuances of a join? Certainly not me.
  - marknadal 13y ago
    Realistically I won't have cost-comparisons for a while, as this project is just in the infancy and obviously needs to stabilize/mature first. Although as a quick summary, your costs should approximately be server(s) that are expensive enough to fit your smallest set of per-user active-data, plus your S3 storage amount, plus S3 API calls (I've made this part a simple options parameter, so you control how often these calls are being made - the more frequent, the more integrity from worst-case disruptions but also more expensive, less frequent and your costs go down). What is worst-case disruption? Everything goes offline simultaneously. User's don't have localstorage fallbacks in their phone/browser so retries rely on the server cache. The server cache is running the default method, not Redis - and then your server crashes or the machine goes under. And finally, of course, there is an S3 outage and you only persisted to 1 region, not multiple regions. I'll get into hairy situations like that in some of my following posts, but too much for here. Yes, you have another good point about query languages. I agree with you very much - somebody else commented about this, check out my reply to muaddirac. Please keep in touch with more questions/comments, or even email me!
opendais 13y ago
Exactly. "...this isn't a database" :/ Distributed caches aren't new either. [e.g. https://github.com/golang/groupcache https://github.com/golang/groupcache ] Can we get a title change to something that labels it a Distributed Cache? [e.g. Shown HN: gun – Massively Distributed Cache ]
- marknadal 13y ago
  opendais, I'm curious if these other distributed caches actually support persistence? I wasn't able to find such a system. But I'm limited in my research, I'd love to learn more about other people's work! Please share more!
  - opendais 13y ago
    http://ehcache.org/ http://ehcache.org/ http://ehcache.org/documentation/configuration/fast-restart#strategy-options http://ehcache.org/documentation/configuration/fast-restart#... Based on a quick googling, this looks like it would be an example of that. Frankly, I don't normally want my cache to persist so it isn't something I know much about. [e.g. I store persistent data in Cassandra or MySQL]
    - marknadal 13y ago
      You hit on a really good point, in the several years that I've worked in the synchronization field (I do consulting) I've found that it has actually been harder to deal with /what you don't want to sync/ than with what you want to sync. It would be really bad if our temporary variables that are used on the client and server to do transformations on our actual data got somehow jumbled up into our real data. I have some clean delineations for this, but wow - I didn't know that this was going to explode so quickly. If I had known, I would have prepared some more resources and materials to explain this stuff. That said, everybody who signs up on the mailing list will get notified about the posts I will write on this! I'm looking forward to getting some serious counter-arguments/refutations to challenge gun.
      - 13y ago

noelwelsh 13y ago

Nice idea which mirrors a lot of my current thinking; far more details are needed. For example: "All conflict resolution happens locally in each peer using a deterministic algorithm." Hrrrmm.... if the model is to use CRDTs for conflict free resolution I can believe it; if it's timestamps, for instance, I'm much more doubtful.

marknadal 13y ago
noelwelsh, thanks for the comment! Timestamps do fail pretty miserably for handling conflict resolution - while my algorithm does use timestamps, the timestamp is hardly responsible for the actual synchronization. It just provides a basis to initially sort things off of, and then the conflict resolution kicks in. Obviously I'm going to need to do a pretty detailed post on how this works, cause unless it can be battled-tested and skeptically investigated by others, it is only as good as snake oil. So more on this soon! Thanks for bringing it up.
- avodonosov 13y ago
  Conflict resolution is the main question.

adambard 13y ago

This actually sounds like a pretty cool and useful project, but I had to force myself to overcome the site, which I think does a poor job of marketing. The headline is a bit confusing -- it's not obvious that something can be "embedded" and "distributed", let alone "massively." You might rewrite it to tell me what the project does, not what it is. Besides that, as a developer, what I want to know is: - Why should I use it? (Instant in-browser storage synced to a distributed server network) - How is it implemented? (it seems to be a javascript project, but that's only mentioned once) - How do I run it? (It run daemons on servers and stores the data in S3) - What do I have to do to configure it? (Seems to me like "The granularity and frequency of these snapshots can be tweaked by you" means "Prepare to get elbow-deep into telling gun how to shard") If you just take the answers to these questions and make them either bullet points up above the video, or headers for your marketing essay, it would help people like me digest the project enough to be interested and read more. Finally, while I'm being a grinch, I'd adjust your styles just slightly to make the site more readable. At the very least: - Make `#main` narrower; perhaps 700px or even less (long lines are hard to read) - In `body`, apply a line-height of maybe 20pt (the lines seem too close together for the font size) - Also in `body`, drop the text-shadow. Shadow on white tends to just make things look blurry. All that said, I am interested in learning more. A self-hosted Firebase would be a real boon for many projects.

marknadal 13y ago
Great suggestions! Already fixed one of the things, and working on updating the others! (Although content will take a while longer). Thank you so much.

reillyse 13y ago

I'm withholding my judgment on the tech - it all seems a little too good to be true - but the copy on your page is great, I actually read every line of it which rarely ever happens when I visit a technical page.

jobeirne 13y ago

I found the header image a bit disconcerting, and immediately hit Back (I'm at work).

marknadal 13y ago
sorry :/ I'm trying to intentionally convey that this tool is dangerously powerful. I don't mean to offend anyone though.
- deleted 13y ago
  [deleted]
- jauer 13y ago
  And here I thought you were showing your good taste in customized 1911 variants... You might want to consider a name that reflects the distributed nature over the dangerous power because of reactions like the parent (although I strongly disagree with the sentiment expressed). Seriously though, this sounds cool and I'm looking forward to trying it out once you release code.
  - marknadal 13y ago
    :P I'm coming to admire guns a lot more because of this library - respect!
- userbinator 13y ago
  I'm not in the "guns are offensive" line of thought but I think it could have a better name than an existing noun. (Although CloudFlare has already taken "railgun"...)
chadillac 13y ago
Pictures of guns are offensive and dangerous at work now? Really?
- TallGuyShort 13y ago
  Heck, I used to work in an office where most people had similar weapons on them at all times.

johne20 13y ago

How would gun handle user auth and data that can only be seen by the authenticated user?

marknadal 13y ago
So I'm making sure that the concepts of piping and transforms are built right into the API, so you'll be able to easily filter out data by whitelists or blacklists as it is getting pushed down to your users. I'll also be writing plugins to gun that will allow people who like ORMs to just attach one on, and the filtering and validation will be handled for them. Does this answer your question?
- johne20 13y ago
  Yes conceptually it does, I would be interested to see how it is implemented. Thanks!

muaddirac 13y ago

Looks interesting! > you never have to learn some silly separate query language again. A query language which just attempts to be some DSL to RPC another machine into doing the same query you could have already written in half the time it took to learn the query language. Not sure I agree with this sentiment - most programming languages aren't declarative like query languages are, and that seems especially useful for, well, querying.

marknadal 13y ago
This is true. The neat thing about the modular design I have for gun is that people can always write a plugin that receives some/any query language, and then translates it directly into the appropriate algorithms. So you should be able to write your own abstractions ontop - but this is only possible because you are able to write the direct queries underneath.

mallyvai 13y ago

"Because face it, any sufficiently capable query language has to be Turing complete" - No, no it doesn't. Stock SQL is not Turing complete. There are extensions that support recursion to make it so, but it is perfectly possible and capable of doing everything you could reasonably want without it. Caching is hard. Consistency is hard. Peer to peer is hard. I hope the author is addressing these in a sane, verifiable way. I'm really curious to see the demo apps that come out of this.

marknadal 13y ago
I apologize, I wasn't saying every query language /is/ Turing complete, just that to do exponentially more complicated tasks in a single atomic query... the lines start to blur between the two. Yes, these are hard subjects - especially things like cache invalidation. You raise good points, and I hope to answer them in the follow up posts I write - I expect your's and other's good eye to check my work. I'll let people know when I've written them. Demos are coming too!

jellicle 13y ago

The name "gun": a) has negative connotations, b) is ungoogleable, c) can't even write about it without being ungrammatical and d) will be filtered out at some workplaces. Dumb, de-dumb dumb dumb.

stewars 13y ago

What license is the code distributed under? It is not clear from the site or the code that is available through npm.

curveship 13y ago

So CAP: which does gun throw off the island? Based on this statement ... > It bridges the distance with a realtime connection, so updates propagate at the speed of the raw pipes linking them. ... I'm assuming it's the P.

dantiberian 13y ago
CA system's don't exist so it must be CP or AP. However, further consideration shows that CA is not really a coherent option because a system that is not Partition- tolerant will, by definition, be forced to give up Consistency or Availability during a partition. https://foundationdb.com/white-papers/the-cap-theorem https://foundationdb.com/white-papers/the-cap-theorem
- GregorStocks 13y ago
  It could also be neither, which is a popular choice these days.
marknadal 13y ago
Out of the box gun only guarantees you eventual consistency, not strong consistency. The reason why is because the majority of web applications out there today are manipulating data sets that aren't particularly significant or regulated (tweets compared to medical records, profile updates compared to payment information). As a result, gun opts for high availability so response times are fast for users, and assuring their data won't be lost. However, the strength of this design is it allows you to implement stronger forms of consistency on top - which is necessary for those occasional pieces of data (payments, medical records, etc). But if you do this then the availability of that data diminishes because servers must wait for consistency to be achieved before doing any further transformations. The sweet spot about this is that it will only be the availability of that particular property (or properties that it depends upon) that decreases, all other pieces of information will still be snappy fast because gun is capable of locking at a granular level. This will not be as "easy" as it is in databases that are strongly consistent to begin with, but those databases aren't capable of being eventually consistent either (because that would violate the DB's principles). But it is possible, and you the developer get the control and flexibility over that. So you win some and you lose some, but I believe this setup is highly optimized because generally speaking the things that you need to be "strongly consistent" anyways have much slower "weakest links" than the speed of networks being up or down - such as approval by some human overseer (a loan officer, a lawyer, a doctor, etc.), and meatspace brains take much longer to process regulatory issues than the potentially couple of hours the network might be down. I'll touch more on these quirks in a separate post after I address conflict resolution.

xkarga00 13y ago

Not everyone is familiar with the concept of NoDB so this paper [1] is a good read to start with. [1] http://bit.ly/RjBl9S http://bit.ly/RjBl9S

Allower 13y ago

I have been looking for something just like this! Fantastic

ilaksh 13y ago

This reminds me a little of Meteor and a bit more of ShareJS. And it reminds me of a few other things. As far as transformations and conflict resolution, I am wondering, are you using something like operational transformations?

marknadal 13y ago
I had to develop my own synchronization algorithms after doing a lot of research on OT for a couple of reasons. One is that Google sometimes relies on being a centralized authority (Google's server) to resolve some conflicts, especially for collaborative rich text editing (Neil Fraser, a sync genius that Google hired, has a great talk on his own type of implementation "diff-match-patch", although Google uses different algorithms now I believe). Another is that OT sometimes requires you writing your own transformation commands, and I wanted something that was generally applicable to most data types without forcing developers to write their own. (It has been a while since I did a lot of this reading, so this comment may not necessarily be accurate/true, please somebody correct me!) Instead, I developed my own method, called "Analytical Fluctuation" - but I haven't written papers on it yet, but need to soon! One requirement is that it has to be truly peer to peer and cannot rely (even occasionally) on some centralized server or leader election setup. Another requirement is that it has to be latent proof, which for me means "preparing for the future" when people need to collaborate on documents not only from the other side of the world but also from Mars. This is where Neil Fraser says his system breaks down, because if there is too much latency, the patches stop applying idempotently. Again, more on this in upcoming posts.

asimpletune 13y ago

Hey, I'm confused by the statement, "No amount of leader election and consensus algorithms can patch this without facing an unjustified amount of complexity. Gun resolves all this by biting the bullet - it solves the hard problems first, not last. It gets data synchronization and conflict resolution right from the beginning, so it never has to rely on vulnerable leader election or consensus locking". My question is, how do you solve data synchronization and conflict resolution, without using the techniques that do that, i.e. leader election, or some other type of consensus algorithm?

marknadal 13y ago
Gun's conflict resolution algorithm is deterministic, meaning that it will choose the same answer on every peer without having to communicate with others which value it chose. This even works when the ordering of incoming updates is switched depending upon which server was the "I" sending updates out to "you" (pronouns, analogously, are inverse of each other depending upon which person is doing the talking). Uh, this is not clear/sounds confusing, I'll explain it better in my post that goes over how the resolution algorithm works. (Basically the gist is that sometimes for two servers to agree on the "same" answer they both have to have an algorithm which results in an inverse condition from what the other would answer - similar to how quantum-entangled particles have opposite spins of each other to "balance out") Point being, this makes gun truly peer to peer, because it behaves correctly if its the only one running, or if there are numerous guns interconnected with each other. No leader election, no consensus algorithm - you don't need those, because the system agrees as soon as the update are received because it resolves them immediately with an idempotent algorithm. Make sense? More details on this soon.
- asimpletune 13y ago
  Then I look forward to your next post. Can you give me a preview on what type of failure model Gun can support? Consensus is easy assuming there are no faults, but if faults are possible, then leadership election, paxos, whatever, something is needed.
  - 13y ago