4 ms·

This title is bad. It's not an embedded database engine. It's not a database at all. The author mentions this directly in the article (How are you different fro

This title is bad. It's not an embedded database engine. It's not a database at all. The author mentions this directly in the article (How are you different from every other database that is trying to reinvent the wheel? 1. Because gun is not a database (NoDB), it is a persisted distributed cache.) There are other possible issues, but this short summary doesn't give nearly enough information about the system to make a possible determination on its efficacy. I will be interested to see further work.

marknadal 13y ago
Hey _dark_matter_, Thanks for the comment! I'll be doing follow up posts soon on the technical details of my conflict resolution algorithms, and will be preparing Jepsen tests for gun. I just wanted to formally announce this now, though, as I have a tendency to overly-perfect stuff, and getting feedback early on is great. Mind listing things you would like me write in-depth posts on? Thanks for the feedback!
- _dark_matter_ 13y ago
  Hey Mark, I'm interested in several different features. First off, you make a great point about "Hosting your own database is a pain" and/or expensive. I'm sure you know about the raging debate that has been happening between MapReduce and Parallel DBs, and a big part of the reason that MapReduce (Hadoop) style has had so much support is because it's easy and cheap. In lots of instances it may run slower, but why pay for the hassle when you can just run Hadoop for free (not including server costs)? You're making the same point here. Whats the threshold on money savings? How much does it cost to use a DaaS (SQL or NoSQL) compared to what you want to do with Gun/Redis? (Also, is this going to be open source?) Next, the conflict resolution should be interesting. What kind of eventual consistency guarantees are you hoping to have? I see you're planning on addressing this. I'm also interested in the querying. That is, are you using Javascript? My problem with what you're saying about current query languages is that it's not really that hard to get the bare usage out of them. Simple queries in SQL are extremely straightforward. In addition, declarative languages really make things easier for the programmer, not harder. Who wants to have to deal with all the nuances of a join? Certainly not me.
  - marknadal 13y ago
    Realistically I won't have cost-comparisons for a while, as this project is just in the infancy and obviously needs to stabilize/mature first. Although as a quick summary, your costs should approximately be server(s) that are expensive enough to fit your smallest set of per-user active-data, plus your S3 storage amount, plus S3 API calls (I've made this part a simple options parameter, so you control how often these calls are being made - the more frequent, the more integrity from worst-case disruptions but also more expensive, less frequent and your costs go down). What is worst-case disruption? Everything goes offline simultaneously. User's don't have localstorage fallbacks in their phone/browser so retries rely on the server cache. The server cache is running the default method, not Redis - and then your server crashes or the machine goes under. And finally, of course, there is an S3 outage and you only persisted to 1 region, not multiple regions. I'll get into hairy situations like that in some of my following posts, but too much for here. Yes, you have another good point about query languages. I agree with you very much - somebody else commented about this, check out my reply to muaddirac. Please keep in touch with more questions/comments, or even email me!
opendais 13y ago
Exactly. "...this isn't a database" :/ Distributed caches aren't new either. [e.g. https://github.com/golang/groupcache https://github.com/golang/groupcache ] Can we get a title change to something that labels it a Distributed Cache? [e.g. Shown HN: gun – Massively Distributed Cache ]
- marknadal 13y ago
  opendais, I'm curious if these other distributed caches actually support persistence? I wasn't able to find such a system. But I'm limited in my research, I'd love to learn more about other people's work! Please share more!
  - opendais 13y ago
    http://ehcache.org/ http://ehcache.org/ http://ehcache.org/documentation/configuration/fast-restart#strategy-options http://ehcache.org/documentation/configuration/fast-restart#... Based on a quick googling, this looks like it would be an example of that. Frankly, I don't normally want my cache to persist so it isn't something I know much about. [e.g. I store persistent data in Cassandra or MySQL]
    - marknadal 13y ago
      You hit on a really good point, in the several years that I've worked in the synchronization field (I do consulting) I've found that it has actually been harder to deal with /what you don't want to sync/ than with what you want to sync. It would be really bad if our temporary variables that are used on the client and server to do transformations on our actual data got somehow jumbled up into our real data. I have some clean delineations for this, but wow - I didn't know that this was going to explode so quickly. If I had known, I would have prepared some more resources and materials to explain this stuff. That said, everybody who signs up on the mailing list will get notified about the posts I will write on this! I'm looking forward to getting some serious counter-arguments/refutations to challenge gun.
      - 13y ago

marknadal 13y ago

Hey _dark_matter_, Thanks for the comment! I'll be doing follow up posts soon on the technical details of my conflict resolution algorithms, and will be preparing Jepsen tests for gun. I just wanted to formally announce this now, though, as I have a tendency to overly-perfect stuff, and getting feedback early on is great. Mind listing things you would like me write in-depth posts on? Thanks for the feedback!

_dark_matter_ 13y ago
Hey Mark, I'm interested in several different features. First off, you make a great point about "Hosting your own database is a pain" and/or expensive. I'm sure you know about the raging debate that has been happening between MapReduce and Parallel DBs, and a big part of the reason that MapReduce (Hadoop) style has had so much support is because it's easy and cheap. In lots of instances it may run slower, but why pay for the hassle when you can just run Hadoop for free (not including server costs)? You're making the same point here. Whats the threshold on money savings? How much does it cost to use a DaaS (SQL or NoSQL) compared to what you want to do with Gun/Redis? (Also, is this going to be open source?) Next, the conflict resolution should be interesting. What kind of eventual consistency guarantees are you hoping to have? I see you're planning on addressing this. I'm also interested in the querying. That is, are you using Javascript? My problem with what you're saying about current query languages is that it's not really that hard to get the bare usage out of them. Simple queries in SQL are extremely straightforward. In addition, declarative languages really make things easier for the programmer, not harder. Who wants to have to deal with all the nuances of a join? Certainly not me.
- marknadal 13y ago
  Realistically I won't have cost-comparisons for a while, as this project is just in the infancy and obviously needs to stabilize/mature first. Although as a quick summary, your costs should approximately be server(s) that are expensive enough to fit your smallest set of per-user active-data, plus your S3 storage amount, plus S3 API calls (I've made this part a simple options parameter, so you control how often these calls are being made - the more frequent, the more integrity from worst-case disruptions but also more expensive, less frequent and your costs go down). What is worst-case disruption? Everything goes offline simultaneously. User's don't have localstorage fallbacks in their phone/browser so retries rely on the server cache. The server cache is running the default method, not Redis - and then your server crashes or the machine goes under. And finally, of course, there is an S3 outage and you only persisted to 1 region, not multiple regions. I'll get into hairy situations like that in some of my following posts, but too much for here. Yes, you have another good point about query languages. I agree with you very much - somebody else commented about this, check out my reply to muaddirac. Please keep in touch with more questions/comments, or even email me!

opendais 13y ago

Exactly. "...this isn't a database" :/ Distributed caches aren't new either. [e.g. https://github.com/golang/groupcache https://github.com/golang/groupcache ] Can we get a title change to something that labels it a Distributed Cache? [e.g. Shown HN: gun – Massively Distributed Cache ]

marknadal 13y ago
opendais, I'm curious if these other distributed caches actually support persistence? I wasn't able to find such a system. But I'm limited in my research, I'd love to learn more about other people's work! Please share more!
- opendais 13y ago
  http://ehcache.org/ http://ehcache.org/ http://ehcache.org/documentation/configuration/fast-restart#strategy-options http://ehcache.org/documentation/configuration/fast-restart#... Based on a quick googling, this looks like it would be an example of that. Frankly, I don't normally want my cache to persist so it isn't something I know much about. [e.g. I store persistent data in Cassandra or MySQL]
  - marknadal 13y ago
    You hit on a really good point, in the several years that I've worked in the synchronization field (I do consulting) I've found that it has actually been harder to deal with /what you don't want to sync/ than with what you want to sync. It would be really bad if our temporary variables that are used on the client and server to do transformations on our actual data got somehow jumbled up into our real data. I have some clean delineations for this, but wow - I didn't know that this was going to explode so quickly. If I had known, I would have prepared some more resources and materials to explain this stuff. That said, everybody who signs up on the mailing list will get notified about the posts I will write on this! I'm looking forward to getting some serious counter-arguments/refutations to challenge gun.
    - 13y ago