7 ms·

Clarification, I said GUN is NOT acid compliant from your "usual" understanding of the term, since GUN is AP. Most people assume acid means CP. ACID is very va

Clarification, I said GUN is NOT acid compliant from your "usual" understanding of the term, since GUN is AP. Most people assume acid means CP. ACID is very vague though, and I'd like to explore it more by writing tests to either confirm or deny whether GUN supports it or not (would you be interested in helping build those tests?). I also want to get some Jepsen like tests up as well. Data convergence (data sync) is guaranteed by the Hypothetical Amnesia Machine algorithm, which is completely deterministic and idempotent. There is some details on it in the wiki, let me know if you have any questions. I also did a tech talk on it. In NO way does gun rely on S3 for consistency. That would be horrible. Check out the algorithm and slam me with questions/critiques. Thanks for looking. :)

theseoafs 12y ago
You do not prove a system is ACID by writing and running test cases. Does the team behind this have experience with distributed systems?
- marknadal 12y ago
  Yes, I've spent the last 4 years doing consulting work and research on them. Warning: I have a very different approach though, more on the side of bittorrent and bitcoin, than what you are going to find in your traditional databases (CP, Master-Slave, Consensus/PAXOS/etc.). If you are a distributed systems person also, I'd really like to talk. If you're armchair/backseat scoffing, then I would still love to talk show you how the algorithms work.
  - theseoafs 12y ago
    Sorry if you were offended by my question; I didn't intend any disrespect, but I hope you'll forgive me if I say that this HN submission is very confusing. The webpage claims that Gun is both the "easiest database ever" and "not a database" (interesting, then, that the website's URL is gundb). It claims that the problem with databases is that they assume there's a "centralized authority", which is sort of absurd; the overwhelming majority of businesses and organizations that use databases have at least some data that they need to absolutely 100% guarantee is safe and consistent, and distributed algorithms with leaders are the easiest way to capture that. Also, what is "vulnerable" about consensus algorithms? Does your distributed database really have no way to reach consensus? Persistence is solved with "any S3 like service"? So what does that mean, using Gun is going to tie me to another unrelated Database as a Service that I'm going to have to pay Amazon for? I'm sure this tool offers something interesting that other tools can't match, since you made it and put time into it, but the existing documentation isn't capturing that yet. Write up another blog post that describes the details, the use cases, the guarantees, etc.; i.e. the actual hard technical details, rather than the PR-speak, and I'll happily take a second look.
    - marknadal 12y ago
      Yes, I got caught red-handed with my NoDB/gunDB marketing speak. Pretty embarrassing, but the point is that it is a distributed persisted cache, so you get the benefits of a DB without having to manage or maintain a DB. You are right, most businesses that run that type of logic probably have the money to afford configuring master-slave based systems. They probably should not move over to GUN. No, you do not have to use S3 for persistence, you can also use your disk. Persistence is a plugin in GUN, so you could build your own module that uses anything to store data. However, there are lots of interesting advantages to distributed/decentralized master-master systems. And I'm trying to make those algorithms available to common man. Thanks for the encouragement, I'll be adding more docs and blogs and stuff. Edit: This person (in the comments below, please upvote him), and my reply, best addresses the most important questions: https://news.ycombinator.com/item?id=9077969 https://news.ycombinator.com/item?id=9077969
fidotron 12y ago
Is this based on CRDTs? ( https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_... . . . or ideas similar to that).
- marknadal 12y ago
  Yes, it is very similar. I'll actually be building some CRDT plugins ontop of GUN core. CRDTs usually deal with specific data types. Interested in helping?
qqueue 12y ago
Is the Hypothetical Amnesia Machine algorithm backed by any scholarly research, or can you at least cite some papers with similar techniques? Blog posts and javascript are nice, but I have a certain fondness for LaTeX-generated PDFs whenever data integrity is involved, e.g. HyperDex's pretty excellent papers: http://hyperdex.org/papers/ http://hyperdex.org/papers/
- marknadal 12y ago
  No papers yet, but I've been working on building up connection with academics and hiring them. So hopefully expect to see something published, but the process takes a while. Meanwhile I'm actively working on building towards a simulation system and an actual high-scale deployable battle testing environment. Think of these as taking the theory from scholastic research, and actually implementing them in practical settings that anybody can run. I'd also like to get a TLA+ specification going. If you have any experience in this stuff, please please please contact me mark@gunDB.io because this is important to me.
  - roeme 12y ago
    When I read your responses, I can‘t shake the feeling I'm talking to some snake oil salesman in a nice suit.¹ And to be honest, GUN‘s docs sound similar. Heavy on how to use, and how awesome everything is, but as soon as one tries to understand stuff, it‘s either WIP or “team up with me/us!”. eyebrow rises And the claim of “building up connection with academics and hiring them” falls perfectly in line with this. Why the hell can‘t you describe what you did by yourself? If it's so awesome, why don‘t you just die to explain it to everyone who asks? Or, $DEITY forbid, should the “academics” lend some credibility to GUN, even if it's with just their title? What's this HAM about? Maybe it's just me, but all this with a rather complex naming convention (souls...) and code... eh, I'll go with “show us teh algoz”. Or describe it. ¹) To illustrate: « I'm actively working [...]» – I'd like to see you passively working.
    - karlgrz 12y ago
      It's not just you.
barakm 12y ago
The problem with AP is that you completely lack consistency guarantees, and most people will agree that's a bad thing. That you advocate that notion so heavily, and without scholarly research makes me really, really nervous. From what I'm seeing in the function you reference (with very few details in the wiki) there's no accounting for, well, `P` -- what happens when a message does not make it, which happens all the time. What I see is sort of, if you squint, a vector sequence, but one that takes nothing about actual distributed systems (and their unreliability) into account. It also completely trusts no errors have happened in states it's receiving. This all through some really messy code... You also claim to not use clocks, but the first thing you do depends on a clock: ``` var serverState = Gun.time.is(); var incomingValue = Gun.is.soul(deltaValue) || deltaValue; var currentValue = Gun.is.soul(current[field]) || current[field]; var state = HAM(serverState, ... ``` Where Gun.time.is() does in fact call current time ``` Util.time.is = function(t){ return t? t instanceof Date : (+new Date().getTime()) } ``` So one thing that will break you right away will be clock skew. That's a critique I get simply and for free. I also see nothing about guaranteeing that states applying to each other are actually what they expect to be. Which, you know, would depend on consistency.... I'm very, very skeptical.
- marknadal 12y ago
  The conflict resolution guarantees eventual consistency, but not strong consistency. Why did I choose that route? Globally locking data is something the universe doesn't even do, we are bound by the laws of physics and can't get faster than the Speed of Light to transmit information. So you have to make a trade off. Now, the cool thing is that you can use AP systems to creating globally locking/strong consistency behavior ontop. I do plan for there to be some plugins/modules that handle this for you, as long as you are aware you're moving into either a very potentially SLOW system, or a centralized one. Also, CRDTs are something that are really important to look into. They're a good reason to try parting ways with CP systems, because they have some fantastic guarantees on data integrity. We're working (as I previously mentioned) on getting academics and papers involved and written. I don't have that much funding yet, though as it is a very expensive and long winded process, but very important. Next issue: LARGE DISCLAIMER: GUN is NOT relying on TIMESTAMPS alone to converge data. THE ACTUAL ALGORITHM is a state machine operating within a boundary function, the boundaries are defined by sort values, which I then use a combination of vector clocks and timestamps. But it is not timestamps alone, timestamps have massive vulnerabilities to them. CLOCK SKEW will NOT break data sync. I've demonstrated syncing happen across machines with bad drift. But you are very right: I need to get docs and videos and evidence of this up ASAP. Pardon for the caps, I just want people to skim and know for sure that I am not ignorant of these things. I'm not trying to sound angry. I'm super happy people care about these things.

theseoafs 12y ago

You do not prove a system is ACID by writing and running test cases. Does the team behind this have experience with distributed systems?

marknadal 12y ago
Yes, I've spent the last 4 years doing consulting work and research on them. Warning: I have a very different approach though, more on the side of bittorrent and bitcoin, than what you are going to find in your traditional databases (CP, Master-Slave, Consensus/PAXOS/etc.). If you are a distributed systems person also, I'd really like to talk. If you're armchair/backseat scoffing, then I would still love to talk show you how the algorithms work.
- theseoafs 12y ago
  Sorry if you were offended by my question; I didn't intend any disrespect, but I hope you'll forgive me if I say that this HN submission is very confusing. The webpage claims that Gun is both the "easiest database ever" and "not a database" (interesting, then, that the website's URL is gundb). It claims that the problem with databases is that they assume there's a "centralized authority", which is sort of absurd; the overwhelming majority of businesses and organizations that use databases have at least some data that they need to absolutely 100% guarantee is safe and consistent, and distributed algorithms with leaders are the easiest way to capture that. Also, what is "vulnerable" about consensus algorithms? Does your distributed database really have no way to reach consensus? Persistence is solved with "any S3 like service"? So what does that mean, using Gun is going to tie me to another unrelated Database as a Service that I'm going to have to pay Amazon for? I'm sure this tool offers something interesting that other tools can't match, since you made it and put time into it, but the existing documentation isn't capturing that yet. Write up another blog post that describes the details, the use cases, the guarantees, etc.; i.e. the actual hard technical details, rather than the PR-speak, and I'll happily take a second look.
  - marknadal 12y ago
    Yes, I got caught red-handed with my NoDB/gunDB marketing speak. Pretty embarrassing, but the point is that it is a distributed persisted cache, so you get the benefits of a DB without having to manage or maintain a DB. You are right, most businesses that run that type of logic probably have the money to afford configuring master-slave based systems. They probably should not move over to GUN. No, you do not have to use S3 for persistence, you can also use your disk. Persistence is a plugin in GUN, so you could build your own module that uses anything to store data. However, there are lots of interesting advantages to distributed/decentralized master-master systems. And I'm trying to make those algorithms available to common man. Thanks for the encouragement, I'll be adding more docs and blogs and stuff. Edit: This person (in the comments below, please upvote him), and my reply, best addresses the most important questions: https://news.ycombinator.com/item?id=9077969 https://news.ycombinator.com/item?id=9077969

fidotron 12y ago

Is this based on CRDTs? ( https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_... . . . or ideas similar to that).

marknadal 12y ago
Yes, it is very similar. I'll actually be building some CRDT plugins ontop of GUN core. CRDTs usually deal with specific data types. Interested in helping?

qqueue 12y ago

Is the Hypothetical Amnesia Machine algorithm backed by any scholarly research, or can you at least cite some papers with similar techniques? Blog posts and javascript are nice, but I have a certain fondness for LaTeX-generated PDFs whenever data integrity is involved, e.g. HyperDex's pretty excellent papers: http://hyperdex.org/papers/ http://hyperdex.org/papers/

marknadal 12y ago
No papers yet, but I've been working on building up connection with academics and hiring them. So hopefully expect to see something published, but the process takes a while. Meanwhile I'm actively working on building towards a simulation system and an actual high-scale deployable battle testing environment. Think of these as taking the theory from scholastic research, and actually implementing them in practical settings that anybody can run. I'd also like to get a TLA+ specification going. If you have any experience in this stuff, please please please contact me mark@gunDB.io because this is important to me.
- roeme 12y ago
  When I read your responses, I can‘t shake the feeling I'm talking to some snake oil salesman in a nice suit.¹ And to be honest, GUN‘s docs sound similar. Heavy on how to use, and how awesome everything is, but as soon as one tries to understand stuff, it‘s either WIP or “team up with me/us!”. eyebrow rises And the claim of “building up connection with academics and hiring them” falls perfectly in line with this. Why the hell can‘t you describe what you did by yourself? If it's so awesome, why don‘t you just die to explain it to everyone who asks? Or, $DEITY forbid, should the “academics” lend some credibility to GUN, even if it's with just their title? What's this HAM about? Maybe it's just me, but all this with a rather complex naming convention (souls...) and code... eh, I'll go with “show us teh algoz”. Or describe it. ¹) To illustrate: « I'm actively working [...]» – I'd like to see you passively working.
  - karlgrz 12y ago
    It's not just you.

barakm 12y ago

The problem with AP is that you completely lack consistency guarantees, and most people will agree that's a bad thing. That you advocate that notion so heavily, and without scholarly research makes me really, really nervous. From what I'm seeing in the function you reference (with very few details in the wiki) there's no accounting for, well, `P` -- what happens when a message does not make it, which happens all the time. What I see is sort of, if you squint, a vector sequence, but one that takes nothing about actual distributed systems (and their unreliability) into account. It also completely trusts no errors have happened in states it's receiving. This all through some really messy code... You also claim to not use clocks, but the first thing you do depends on a clock: ``` var serverState = Gun.time.is(); var incomingValue = Gun.is.soul(deltaValue) || deltaValue; var currentValue = Gun.is.soul(current[field]) || current[field]; var state = HAM(serverState, ... ``` Where Gun.time.is() does in fact call current time ``` Util.time.is = function(t){ return t? t instanceof Date : (+new Date().getTime()) } ``` So one thing that will break you right away will be clock skew. That's a critique I get simply and for free. I also see nothing about guaranteeing that states applying to each other are actually what they expect to be. Which, you know, would depend on consistency.... I'm very, very skeptical.

marknadal 12y ago
The conflict resolution guarantees eventual consistency, but not strong consistency. Why did I choose that route? Globally locking data is something the universe doesn't even do, we are bound by the laws of physics and can't get faster than the Speed of Light to transmit information. So you have to make a trade off. Now, the cool thing is that you can use AP systems to creating globally locking/strong consistency behavior ontop. I do plan for there to be some plugins/modules that handle this for you, as long as you are aware you're moving into either a very potentially SLOW system, or a centralized one. Also, CRDTs are something that are really important to look into. They're a good reason to try parting ways with CP systems, because they have some fantastic guarantees on data integrity. We're working (as I previously mentioned) on getting academics and papers involved and written. I don't have that much funding yet, though as it is a very expensive and long winded process, but very important. Next issue: LARGE DISCLAIMER: GUN is NOT relying on TIMESTAMPS alone to converge data. THE ACTUAL ALGORITHM is a state machine operating within a boundary function, the boundaries are defined by sort values, which I then use a combination of vector clocks and timestamps. But it is not timestamps alone, timestamps have massive vulnerabilities to them. CLOCK SKEW will NOT break data sync. I've demonstrated syncing happen across machines with bad drift. But you are very right: I need to get docs and videos and evidence of this up ASAP. Pardon for the caps, I just want people to skim and know for sure that I am not ignorant of these things. I'm not trying to sound angry. I'm super happy people care about these things.