3 ms·

The problem with AP is that you completely lack consistency guarantees, and most people will agree that's a bad thing. That you advocate that notion so heavily,

by barakm 12y ago

The problem with AP is that you completely lack consistency guarantees, and most people will agree that's a bad thing. That you advocate that notion so heavily, and without scholarly research makes me really, really nervous. From what I'm seeing in the function you reference (with very few details in the wiki) there's no accounting for, well, `P` -- what happens when a message does not make it, which happens all the time. What I see is sort of, if you squint, a vector sequence, but one that takes nothing about actual distributed systems (and their unreliability) into account. It also completely trusts no errors have happened in states it's receiving. This all through some really messy code... You also claim to not use clocks, but the first thing you do depends on a clock: ``` var serverState = Gun.time.is(); var incomingValue = Gun.is.soul(deltaValue) || deltaValue; var currentValue = Gun.is.soul(current[field]) || current[field]; var state = HAM(serverState, ... ``` Where Gun.time.is() does in fact call current time ``` Util.time.is = function(t){ return t? t instanceof Date : (+new Date().getTime()) } ``` So one thing that will break you right away will be clock skew. That's a critique I get simply and for free. I also see nothing about guaranteeing that states applying to each other are actually what they expect to be. Which, you know, would depend on consistency.... I'm very, very skeptical.

marknadal 12y ago
The conflict resolution guarantees eventual consistency, but not strong consistency. Why did I choose that route? Globally locking data is something the universe doesn't even do, we are bound by the laws of physics and can't get faster than the Speed of Light to transmit information. So you have to make a trade off. Now, the cool thing is that you can use AP systems to creating globally locking/strong consistency behavior ontop. I do plan for there to be some plugins/modules that handle this for you, as long as you are aware you're moving into either a very potentially SLOW system, or a centralized one. Also, CRDTs are something that are really important to look into. They're a good reason to try parting ways with CP systems, because they have some fantastic guarantees on data integrity. We're working (as I previously mentioned) on getting academics and papers involved and written. I don't have that much funding yet, though as it is a very expensive and long winded process, but very important. Next issue: LARGE DISCLAIMER: GUN is NOT relying on TIMESTAMPS alone to converge data. THE ACTUAL ALGORITHM is a state machine operating within a boundary function, the boundaries are defined by sort values, which I then use a combination of vector clocks and timestamps. But it is not timestamps alone, timestamps have massive vulnerabilities to them. CLOCK SKEW will NOT break data sync. I've demonstrated syncing happen across machines with bad drift. But you are very right: I need to get docs and videos and evidence of this up ASAP. Pardon for the caps, I just want people to skim and know for sure that I am not ignorant of these things. I'm not trying to sound angry. I'm super happy people care about these things.
- ahelwer 12y ago
  What is a boundary function?
  - marknadal 12y ago
    This person (in the comments below, please upvote him), and my reply, best addresses the most important questions (including the boundary function): https://news.ycombinator.com/item?id=9077969 https://news.ycombinator.com/item?id=9077969
- barakm 12y ago
  So I tried it. I successfully lost data, and reported conflicting/inconsistent data, with a one-line change to your example to-do list. Instead of generating a key that won't conflict, I force it to write to the same key. Yes, this means it's one list item. That's fine, I just need to make this list item inconsistent between two machines. And then I simulate a network partition (virtual machine network disconnect. pull the virtual plug. (EDIT: and repair it!)) This is enough to cause a seconds-long lag before the "okay" case comes through. And a local write during that lag? Completely lost. Two windows. Virtual machine. Game over. This is because your data is overwriting itself, and I don't even care which version I got. I just needed it to not (locally) be the correct one, for any amount of time. It effectively becomes "last write wins" and, well, there's nothing new under the sun. Curiouser and curiouser, the "last write wins" happens in about five seconds. Which happens to be approx. the clock skew. (Called it.) There's some timestamp reliance for sure. It's really easy to break. Caching layer, again, perhaps. Database? Nope. From some people who actually know what they're talking about: http://basho.com/clocks-are-bad-or-welcome-to-distributed-systems/ http://basho.com/clocks-are-bad-or-welcome-to-distributed-sy... (I also got exactly the overwriting "always win" behavior you discussed in the other comment as being a flaw of vector clocks. From where I'm sitting, this smells like a pretty garden-variety vector clock.)
  - marknadal 12y ago
    Thank you for trying it! You should also post this to the Issues on the repo. I'd like to get some more details though... On Keys: keys are unique, meaning they can only point to one thing. When you "forced" it to write to the same key, you changed references, your previous data is still stored, but you'll only be able to find it by looking it up by its soul (its ID) or by scanning for it. Were you experiencing something different than what I just mentioned above? On Network Partition: The clients store their updates in LocalStorage before they send the peers the update or get an ACK. What caused those peers (tabs) to lose their data? For clock skew, please see this reply: https://news.ycombinator.com/item?id=9077969 https://news.ycombinator.com/item?id=9077969 . Most "last write wins" algorithms require a centralized server (a Single Source of Truth) or they diverge on separate machines. That doesn't happen with GUN, you have no SST and they converge (even if it "looks" like last write wins) on both machines in an eventually consistent manner. I'd like to follow up with these things, either here, over email mark@gunDB.io, on the gitter chat, or on github.

by barakm 12y ago

marknadal 12y ago

The conflict resolution guarantees eventual consistency, but not strong consistency. Why did I choose that route? Globally locking data is something the universe doesn't even do, we are bound by the laws of physics and can't get faster than the Speed of Light to transmit information. So you have to make a trade off. Now, the cool thing is that you can use AP systems to creating globally locking/strong consistency behavior ontop. I do plan for there to be some plugins/modules that handle this for you, as long as you are aware you're moving into either a very potentially SLOW system, or a centralized one. Also, CRDTs are something that are really important to look into. They're a good reason to try parting ways with CP systems, because they have some fantastic guarantees on data integrity. We're working (as I previously mentioned) on getting academics and papers involved and written. I don't have that much funding yet, though as it is a very expensive and long winded process, but very important. Next issue: LARGE DISCLAIMER: GUN is NOT relying on TIMESTAMPS alone to converge data. THE ACTUAL ALGORITHM is a state machine operating within a boundary function, the boundaries are defined by sort values, which I then use a combination of vector clocks and timestamps. But it is not timestamps alone, timestamps have massive vulnerabilities to them. CLOCK SKEW will NOT break data sync. I've demonstrated syncing happen across machines with bad drift. But you are very right: I need to get docs and videos and evidence of this up ASAP. Pardon for the caps, I just want people to skim and know for sure that I am not ignorant of these things. I'm not trying to sound angry. I'm super happy people care about these things.

ahelwer 12y ago
What is a boundary function?
- marknadal 12y ago
  This person (in the comments below, please upvote him), and my reply, best addresses the most important questions (including the boundary function): https://news.ycombinator.com/item?id=9077969 https://news.ycombinator.com/item?id=9077969
barakm 12y ago
So I tried it. I successfully lost data, and reported conflicting/inconsistent data, with a one-line change to your example to-do list. Instead of generating a key that won't conflict, I force it to write to the same key. Yes, this means it's one list item. That's fine, I just need to make this list item inconsistent between two machines. And then I simulate a network partition (virtual machine network disconnect. pull the virtual plug. (EDIT: and repair it!)) This is enough to cause a seconds-long lag before the "okay" case comes through. And a local write during that lag? Completely lost. Two windows. Virtual machine. Game over. This is because your data is overwriting itself, and I don't even care which version I got. I just needed it to not (locally) be the correct one, for any amount of time. It effectively becomes "last write wins" and, well, there's nothing new under the sun. Curiouser and curiouser, the "last write wins" happens in about five seconds. Which happens to be approx. the clock skew. (Called it.) There's some timestamp reliance for sure. It's really easy to break. Caching layer, again, perhaps. Database? Nope. From some people who actually know what they're talking about: http://basho.com/clocks-are-bad-or-welcome-to-distributed-systems/ http://basho.com/clocks-are-bad-or-welcome-to-distributed-sy... (I also got exactly the overwriting "always win" behavior you discussed in the other comment as being a flaw of vector clocks. From where I'm sitting, this smells like a pretty garden-variety vector clock.)
- marknadal 12y ago
  Thank you for trying it! You should also post this to the Issues on the repo. I'd like to get some more details though... On Keys: keys are unique, meaning they can only point to one thing. When you "forced" it to write to the same key, you changed references, your previous data is still stored, but you'll only be able to find it by looking it up by its soul (its ID) or by scanning for it. Were you experiencing something different than what I just mentioned above? On Network Partition: The clients store their updates in LocalStorage before they send the peers the update or get an ACK. What caused those peers (tabs) to lose their data? For clock skew, please see this reply: https://news.ycombinator.com/item?id=9077969 https://news.ycombinator.com/item?id=9077969 . Most "last write wins" algorithms require a centralized server (a Single Source of Truth) or they diverge on separate machines. That doesn't happen with GUN, you have no SST and they converge (even if it "looks" like last write wins) on both machines in an eventually consistent manner. I'd like to follow up with these things, either here, over email mark@gunDB.io, on the gitter chat, or on github.