14 ms·

sec MIT Licensed Graph Database

142 points by marknadal 9y ago

mosselman 9y ago
I checked GUN out a while back and it looked cool, but I wanted something to use with react-native. Now there seems to be a package to do this (https://github.com/staltz/gun-asyncstorage https://github.com/staltz/gun-asyncstorage), but I am still unsure about how production ready this is. Any thoughts in general or experiences with Gun and react-native?
- marknadal 9y ago
  Yes, Andre Staltz's (of CycleJS) work finally got it working on Android and iOS with React-Native! We're planning on having an example/starter app for it soon (unfortunately, Google Chrome and iOS follow the WebSocket spec just slightly differently, so it errors on iOS currently but it is an easy fix, we just need to figure out a way for them to both work simultaneously).
  - mosselman 9y ago
    That sounds great :). Any place I should keep an eye on for updates?
    - marknadal 9y ago
      The https://gitter.im/amark/gun https://gitter.im/amark/gun is the best place, but it is also pretty noisy with chatter. Else you can probably just keep your eye on Andre's twitter, he's posted his other stuff there. Anywho, would love to learn what you are working on! Shoot me an email or something. :)
ddorian43 9y ago
Come on man, do we have to do this everytime ? Next time post on a weekday so you'll get maximum criticism. They should've implemented scylladb on top gundb, not amateur-designed-by-kernel-hackers seastar-framework. Do reavaluate your time, seriously & sincerly.
- deleted 9y ago
  [deleted]
falcolas 9y ago
That collision resolution algorithm looks like a doozie from a "malicious peer" point of view. Updates in the past are "recorded and discarded", updates in the future queue up. First, let's see if we can run our peers out of memory with a few (billion) quick future updates. Funny thing, gzip; it's so easy to compress highly repetitive patterns. Could also just try and spam the historical log too, could we fill the disk as well as working memory? If we don't run everything out of memory, let's just write out a few billion updates for every state interval, and make sure it evaluates as "greater than" (yay, JavaScript) any real value. Those updates should preemptively overwrite every other update that comes in. Do you have an operating window of less than, say, 300ms? Heaven help the poor client from Sidney who keeps trying to update a master in London. Their updates will always be discarded (I'm not certain, but the docs read as if this occurs even when updating a value which hasn't been overwritten by a future state). Darn you, speed of light; why can't you be just a little faster? I guess you can only hope that your clients all decide to be honest and never change your code. Or get their state counter (or clock) too far out of sync.
- marknadal 9y ago
  This is a really great comment, thank you - hopefully I can address some of the points you brought up: - LRU/GC hasn't been added (planned for v0.8.x), so you are correct sending in a bunch of updates will crash the peer currently. The "out of bound" updates (for everybody else not sure what the parent is referencing, see this tech talk where I explain what is going on: http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html ) are volatile (this is intentional) because they are considered potentially malicious, so upon crash they'll be lost as it is the origin peer's responsibility to retry updates (and gun automatically does) until ACKs have been received. Thankfully, updates that are "within bounds" will be kept safe. - However, you are right (in your "If we don't...") "within bound" updates may also be malicious. GUN's base algorithm is designed to work in an entirely ad-hoc anonymous mesh network. To deal with this, we just recently announced our Security, Encryption, and Authorization framework to handle trusted peers, here: https://github.com/amark/gun/wiki/auth https://github.com/amark/gun/wiki/auth - GUN is master-master, so no, even with a sliding window of 300ms from latency shouldn't effect the data. Your Sydney to London example is good, I've played a live action game built on top of GUN from Australia <-> USA, and even with P2P logic (no master server) it is responsive. The whole space ship game is only 190 LOC and you can play it here: http://gunjs.herokuapp.com/game/space.html http://gunjs.herokuapp.com/game/space.html (Warning: it is kinda a lame game, but proof conflicting updates in game states work just fine.) - Also, fun fact, in a Master-Master system, the clock drift/skew on machines from different continents can become quite bad. Since we don't have atomic clocks like Spanner, we had to write a P2P version of NTP that runs along side the game. You can test how well it works across your devices here: http://gunjs.herokuapp.com/game/nts.html http://gunjs.herokuapp.com/game/nts.html - Additionally, very rarely are you going to have that many writes within a 300ms span on the /same/ record. However you very easily might have that many for a Twitter like app. Back a year ago, we ran this load test on a prototype storage driver for work loads like this (append-only) and scaled to 100M+ messages for $10/day (all costs: CPU, disk, backup), check out the proof here: https://www.youtube.com/watch?v=x_WqBuEA7s8 https://www.youtube.com/watch?v=x_WqBuEA7s8 Thank you very much for writing your comment. Did I miss anything? More than happy to address any other concerns. The more we can challenge the assertions/claims of database vendors (like me), the better the industry will be off. Let me know if I can answer anything else. Thanks again!
marknadal 9y ago
Some people asked why this release is so important: - It fixes several critical bugs that happened during the performance rewrite. Example: If a server crashed and had its data wiped, there wound up being some sync issues. But this release fixes those. - First time for us to hit 2K table inserts/second synced end-to-end across a federated (browser <-> server <-> server <-> browser) network topology. This load test was running on low end hardware, so expect better results on better hardware. - These tests are now available for anyone to run, using our distributed testing framework called PANIC, which simulates failure cases (inspired by Aphyr's Jepsen.io tests). Code and docs for it at https://github.com/gundb/panic-server https://github.com/gundb/panic-server . - - If you want to run (or write your own) please read through this well-commented 300LOC test: https://github.com/amark/gun/blob/master/test/panic/load.js https://github.com/amark/gun/blob/master/test/panic/load.js . - - The test that was added in this release simulates what happens to GUN in a split brain network partition during a server loss. We expect the data to converge once the network heals (it previously was not, but now does). The PANIC test for this is here: https://github.com/amark/gun/blob/master/test/panic/holy-grail.js https://github.com/amark/gun/blob/master/test/panic/holy-gra... (Warning: not commented, please see the previous test to understand what is going on) Happy to answer any other questions. For anybody using GUN, this is one of the most important releases and upgrading is strongly recommended.
- jadbox 9y ago
  What are some appropriate real production use-cases for this? Particularly since you can't trust the user to tell you the 'real' state of a resource, the REST model is to send discrete state changes to resources and having an explicit endpoint to validate that the change is allowed (and logical).
  - marknadal 9y ago
    We're currently seeing these use cases: - Autonomous IoT on mission critical hardware. (We rolled out a pilot with a government for this) - Distributed machine learning. (https://github.com/cstefanache/cstefanache.github.io/blob/master/_posts/2016-08-02-gun-db-artificial-knowledge-sharing.md#gundb https://github.com/cstefanache/cstefanache.github.io/blob/ma...) - Games (built with React! https://github.com/PsychoLlama/connect-four https://github.com/PsychoLlama/connect-four ) and realtime GPS tracking (https://youtu.be/7ALHtbC9aOM https://youtu.be/7ALHtbC9aOM). - And we're chatting with a bunch of companies doing some really cool stuff: robotic manufacturing, ethereum/blockchain 'lightning network' state channels, realtime decision engine / predictive fraud detection, federated home servers / DNS on top of gun, multiplayer VR, and more. You should NOT use gun for any globally/strongly consistent data, like bank account balances and such. Ever since we introduced our Security, Encryption, Authorization framework, people are starting to build more user facing apps, like P2P crypto social networks, and collaborative tools. Which leads to.... if you cryptographically sign the data, you can trust that the user's copy of the data! It is very untraditional, but that is what we are trying to push/encourage. We even produced an entire animated explainer series, 1 minute each, to explain the concepts: http://gun.js.org/explainers/data/security.html http://gun.js.org/explainers/data/security.html ! REST is a very good model, I actually designed gun around the best parts of it - GET and PATCH (although for simplicity it is just called PUT). Graphs are a stateless representation of the data, which when statefully connected (via WebSockets or something), allow us to efficiently transfer only the delta/diffs and maintain sync. While you can run a traditional server with gun as an endpoint for validation, it is actually possible to do validation client side (as long as all clients are running the same rules) as well.
avodonosov 9y ago
> ... and then when the network comes back online GUN will automatically synchronize all the changes and handle any conflicts for you. Any conflicts? How?
- marknadal 9y ago
  Very important question! The best explanations that cover how it works, as well as its tradeoffs/weaknesses are here: (Note: You should not use GUN for any data you need strong/global consistency with, like bank account balances and such. There are much better solutions than GUN.) - http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html - https://github.com/amark/gun/wiki/CAP-Theorem https://github.com/amark/gun/wiki/CAP-Theorem - https://github.com/amark/gun/wiki/Conflict-Resolution-with-Guns https://github.com/amark/gun/wiki/Conflict-Resolution-with-G...
mwpmaybe 9y ago
For god's sake, man, run a linter on this thing. The mixing of spaces and tabs alone makes the code embarrassingly difficult to read. http://imgur.com/a/XlSQD http://imgur.com/a/XlSQD
- roblabla 9y ago
  Fix your editor and set the tabspace to the correct value ? I'm guessing 2 spaces is the right value in this case.
  - mapimopi 9y ago
    Or maybe it's the developer who needs to fix his editor. Proper editor should convert tabs into spaces or vise versa and not create complications for everyone else.
  - mwpmaybe 9y ago
    Last I checked I don't have access to "fix" how GitHub displays source code, although it looks like the maintainers can add an .editorconfig to their project to address this particular case.
ccommsxx 9y ago
We're getting even better numbers on other devices: Android phone, ~5M ops/sec. Macbook Air, Chrome, ~30M ops/sec. Macbook Pro, Chrome Canary, ~80M ops/sec. Lenovo netbook, IE6, ~100K ops/sec [...] These numbers represent a breakthrough in performance not possible with other databases. Care to share some details on this benchmark? What kind of operations were these? Your results don't really sound likely for any kind of non-trivial operation - is it possible that you were testing a routine that was optimized out (to a static return/noop) by the JIT? from https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-Atom-CPU https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2...
- benwilber0 9y ago
  yeah 5 million "ops"/second on an Android phone? I want to know what the "ops" were. Is it just reading a value from a static memory address? Network request? What's going on here?
- marknadal 9y ago
  JS performance is terrible, it took us a year to rewrite everything from scratch to get these type of numbers. As mentioned near the bottom of that link, they are on cached reads which is unrealistic in real world settings but a useful baseline number to compare against. To understand our methodology on the benchmark approach, check out this tech talk on all the problems we encountered while testing (including things like you mentioned, dealing with how the JIT optimizes, or often times in Chrome, doesn't): https://youtu.be/BEqH-oZ4UXI https://youtu.be/BEqH-oZ4UXI However, a perk of the push-based model (realtime data sync like with Firebase), is that cached reads become a lot more realistic because often the data will already be there before you read it (unlike with a pull/poll based model). Any details I can expand on for you?
  - benwilber0 9y ago
    > Android phone, ~5M ops/sec. what are the "ops" in this case?
    - 9y ago
akanet 9y ago
Do you plan to add presence to GUN? One of the key features that we require from Firebase is the ability to tell the server to delete a key in the event the client disconnects. https://firebase.google.com/docs/reference/android/com/google/firebase/database/OnDisconnect https://firebase.google.com/docs/reference/android/com/googl...
- marknadal 9y ago
  Not yet! Good suggestion though. Deletes right now are currently very difficult in gun. But we plan on fixing that in v0.8.x releases. Your idea of being able to do it on disconnects is a great one, I'll add it to the to-do list!
lacampbell 9y ago
This looks really cool. I'm no database expert, but a decentralised graph database is an intriguing idea. Has it been tried before? (decentralised databases I mean, not graph databases)
- grandalf 9y ago
  I'm building one, slightly different use case though.
  - marknadal 9y ago
    Nice! Have any links to share?
    - grandalf 9y ago
      not yet but perhaps in the future. We're hiring fwiw.
- jhoechtl 9y ago
  https://dgraph.io/ https://dgraph.io/
- vpol 9y ago
  I hope it was a sarcastic comment.
maxdemarzi 9y ago
The docs seem to indicate you do not store properties on relationships? Is that right? Then why call it a graph database when it is really an RDF store?
- marknadal 9y ago
  You can do this, but it is not a built in primitive (like it is in Neo4j). We chose to do this because the majority of edges are usually directed, and we wanted to keep the base system as small as possible (all of gun is only about 12KB). It is easy to create "edge" nodes that store relationship properties, but nobody has created a full feature extension/framework for this yet, but I expect one will be built because this is a very useful feature to have.
deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
jimktrains2 9y ago
The title and the first line of the README > GUN is a realtime, distributed, offline-first, graph database engine. Doing 15M+ ops/sec in just ~12KB gzipped. But no mention of any guarantees provided by the system. What kind of transactions are supported? are writes consensus-based, best effort, eventual consistent? The "GUN Survives a Primary Fault" raises more questions than answers. The "primary" appears to be the only server, and the clients are simply caching un-confirmed updates. Unless every client is a full copy of the database (which would seem prohibitive for the types of things this is aimed at?) there wasn't really any fault tolerance showed here. Also, all clients resolved to the most recent update, the one didn't fail because the row had been updated between the last read. While that's OK for some designs, it just smells fishy. I shouldn't be able to trash someone else's update. I feel like this is yet another datastore trying to call itself a database, but offering no guarentees of any kind that your data is safe.
- marknadal 9y ago
  The README mentions it in the #documentation section and links to these resources: (let me know how they can be improved) - CAP Theorem tradeoffs: https://github.com/amark/gun/wiki/CAP-Theorem https://github.com/amark/gun/wiki/CAP-Theorem (We are AP, which is eventually consistent) - My tech talk of how the CRDT works: http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html (compared to consensus protocols) Clients keep a copy of data that they are interested in (not the whole data set, correct), and together they can reconstruct the entire data set. You can also run multiple server peers that backup the whole data set, or use a shard key to determine which subsets. If this isn't fault tolerant, would you mind giving me some examples of what is? Yes, good point, to prevent data from getting trashed you need to run auth on the system, more info here: https://github.com/amark/gun/wiki/auth https://github.com/amark/gun/wiki/auth We certainly aren't a storage engine, but we are a database/datastore (whatever you want to call it), or more appropriately: an open data sync protocol. I have tried to make our guarantees/tradeoffs very well laid out in the above mentioned articles/README, anything I can do to make this more clear/obvious?
  - jimktrains2 9y ago
anon12345690 9y ago
This poster has been working on this and spamming it all over HN for years, it's been proven to have major flaws and full of misleading claims with all those posts. millions of reads per second on mobile devices for a distributed database? just come on, why even post this?
- marknadal 9y ago
  Throughout the rest of the comments I've tried to provide a bunch of evidence/links for the performance claims (and tests that you can run yourself). Let me know if I missed anything and I'd love to chat about it! But proven flawed? Could you send me some links/evidence on that? I'd love to do my best to reply/address any concerns.

142 points by marknadal 9y ago

I checked GUN out a while back and it looked cool, but I wanted something to use with react-native. Now there seems to be a package to do this (https://github.com/staltz/gun-asyncstorage https://github.com/staltz/gun-asyncstorage), but I am still unsure about how production ready this is. Any thoughts in general or experiences with Gun and react-native?

marknadal 9y ago
Yes, Andre Staltz's (of CycleJS) work finally got it working on Android and iOS with React-Native! We're planning on having an example/starter app for it soon (unfortunately, Google Chrome and iOS follow the WebSocket spec just slightly differently, so it errors on iOS currently but it is an easy fix, we just need to figure out a way for them to both work simultaneously).
- mosselman 9y ago
  That sounds great :). Any place I should keep an eye on for updates?
  - marknadal 9y ago
    The https://gitter.im/amark/gun https://gitter.im/amark/gun is the best place, but it is also pretty noisy with chatter. Else you can probably just keep your eye on Andre's twitter, he's posted his other stuff there. Anywho, would love to learn what you are working on! Shoot me an email or something. :)

ddorian43 9y ago

Come on man, do we have to do this everytime ? Next time post on a weekday so you'll get maximum criticism. They should've implemented scylladb on top gundb, not amateur-designed-by-kernel-hackers seastar-framework. Do reavaluate your time, seriously & sincerly.

deleted 9y ago
[deleted]

falcolas 9y ago

That collision resolution algorithm looks like a doozie from a "malicious peer" point of view. Updates in the past are "recorded and discarded", updates in the future queue up. First, let's see if we can run our peers out of memory with a few (billion) quick future updates. Funny thing, gzip; it's so easy to compress highly repetitive patterns. Could also just try and spam the historical log too, could we fill the disk as well as working memory? If we don't run everything out of memory, let's just write out a few billion updates for every state interval, and make sure it evaluates as "greater than" (yay, JavaScript) any real value. Those updates should preemptively overwrite every other update that comes in. Do you have an operating window of less than, say, 300ms? Heaven help the poor client from Sidney who keeps trying to update a master in London. Their updates will always be discarded (I'm not certain, but the docs read as if this occurs even when updating a value which hasn't been overwritten by a future state). Darn you, speed of light; why can't you be just a little faster? I guess you can only hope that your clients all decide to be honest and never change your code. Or get their state counter (or clock) too far out of sync.

marknadal 9y ago
This is a really great comment, thank you - hopefully I can address some of the points you brought up: - LRU/GC hasn't been added (planned for v0.8.x), so you are correct sending in a bunch of updates will crash the peer currently. The "out of bound" updates (for everybody else not sure what the parent is referencing, see this tech talk where I explain what is going on: http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html ) are volatile (this is intentional) because they are considered potentially malicious, so upon crash they'll be lost as it is the origin peer's responsibility to retry updates (and gun automatically does) until ACKs have been received. Thankfully, updates that are "within bounds" will be kept safe. - However, you are right (in your "If we don't...") "within bound" updates may also be malicious. GUN's base algorithm is designed to work in an entirely ad-hoc anonymous mesh network. To deal with this, we just recently announced our Security, Encryption, and Authorization framework to handle trusted peers, here: https://github.com/amark/gun/wiki/auth https://github.com/amark/gun/wiki/auth - GUN is master-master, so no, even with a sliding window of 300ms from latency shouldn't effect the data. Your Sydney to London example is good, I've played a live action game built on top of GUN from Australia <-> USA, and even with P2P logic (no master server) it is responsive. The whole space ship game is only 190 LOC and you can play it here: http://gunjs.herokuapp.com/game/space.html http://gunjs.herokuapp.com/game/space.html (Warning: it is kinda a lame game, but proof conflicting updates in game states work just fine.) - Also, fun fact, in a Master-Master system, the clock drift/skew on machines from different continents can become quite bad. Since we don't have atomic clocks like Spanner, we had to write a P2P version of NTP that runs along side the game. You can test how well it works across your devices here: http://gunjs.herokuapp.com/game/nts.html http://gunjs.herokuapp.com/game/nts.html - Additionally, very rarely are you going to have that many writes within a 300ms span on the /same/ record. However you very easily might have that many for a Twitter like app. Back a year ago, we ran this load test on a prototype storage driver for work loads like this (append-only) and scaled to 100M+ messages for $10/day (all costs: CPU, disk, backup), check out the proof here: https://www.youtube.com/watch?v=x_WqBuEA7s8 https://www.youtube.com/watch?v=x_WqBuEA7s8 Thank you very much for writing your comment. Did I miss anything? More than happy to address any other concerns. The more we can challenge the assertions/claims of database vendors (like me), the better the industry will be off. Let me know if I can answer anything else. Thanks again!

marknadal 9y ago

Some people asked why this release is so important: - It fixes several critical bugs that happened during the performance rewrite. Example: If a server crashed and had its data wiped, there wound up being some sync issues. But this release fixes those. - First time for us to hit 2K table inserts/second synced end-to-end across a federated (browser <-> server <-> server <-> browser) network topology. This load test was running on low end hardware, so expect better results on better hardware. - These tests are now available for anyone to run, using our distributed testing framework called PANIC, which simulates failure cases (inspired by Aphyr's Jepsen.io tests). Code and docs for it at https://github.com/gundb/panic-server https://github.com/gundb/panic-server . - - If you want to run (or write your own) please read through this well-commented 300LOC test: https://github.com/amark/gun/blob/master/test/panic/load.js https://github.com/amark/gun/blob/master/test/panic/load.js . - - The test that was added in this release simulates what happens to GUN in a split brain network partition during a server loss. We expect the data to converge once the network heals (it previously was not, but now does). The PANIC test for this is here: https://github.com/amark/gun/blob/master/test/panic/holy-grail.js https://github.com/amark/gun/blob/master/test/panic/holy-gra... (Warning: not commented, please see the previous test to understand what is going on) Happy to answer any other questions. For anybody using GUN, this is one of the most important releases and upgrading is strongly recommended.

jadbox 9y ago
What are some appropriate real production use-cases for this? Particularly since you can't trust the user to tell you the 'real' state of a resource, the REST model is to send discrete state changes to resources and having an explicit endpoint to validate that the change is allowed (and logical).
- marknadal 9y ago
  We're currently seeing these use cases: - Autonomous IoT on mission critical hardware. (We rolled out a pilot with a government for this) - Distributed machine learning. (https://github.com/cstefanache/cstefanache.github.io/blob/master/_posts/2016-08-02-gun-db-artificial-knowledge-sharing.md#gundb https://github.com/cstefanache/cstefanache.github.io/blob/ma...) - Games (built with React! https://github.com/PsychoLlama/connect-four https://github.com/PsychoLlama/connect-four ) and realtime GPS tracking (https://youtu.be/7ALHtbC9aOM https://youtu.be/7ALHtbC9aOM). - And we're chatting with a bunch of companies doing some really cool stuff: robotic manufacturing, ethereum/blockchain 'lightning network' state channels, realtime decision engine / predictive fraud detection, federated home servers / DNS on top of gun, multiplayer VR, and more. You should NOT use gun for any globally/strongly consistent data, like bank account balances and such. Ever since we introduced our Security, Encryption, Authorization framework, people are starting to build more user facing apps, like P2P crypto social networks, and collaborative tools. Which leads to.... if you cryptographically sign the data, you can trust that the user's copy of the data! It is very untraditional, but that is what we are trying to push/encourage. We even produced an entire animated explainer series, 1 minute each, to explain the concepts: http://gun.js.org/explainers/data/security.html http://gun.js.org/explainers/data/security.html ! REST is a very good model, I actually designed gun around the best parts of it - GET and PATCH (although for simplicity it is just called PUT). Graphs are a stateless representation of the data, which when statefully connected (via WebSockets or something), allow us to efficiently transfer only the delta/diffs and maintain sync. While you can run a traditional server with gun as an endpoint for validation, it is actually possible to do validation client side (as long as all clients are running the same rules) as well.

avodonosov 9y ago

> ... and then when the network comes back online GUN will automatically synchronize all the changes and handle any conflicts for you. Any conflicts? How?

marknadal 9y ago
Very important question! The best explanations that cover how it works, as well as its tradeoffs/weaknesses are here: (Note: You should not use GUN for any data you need strong/global consistency with, like bank account balances and such. There are much better solutions than GUN.) - http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html - https://github.com/amark/gun/wiki/CAP-Theorem https://github.com/amark/gun/wiki/CAP-Theorem - https://github.com/amark/gun/wiki/Conflict-Resolution-with-Guns https://github.com/amark/gun/wiki/Conflict-Resolution-with-G...

mwpmaybe 9y ago

For god's sake, man, run a linter on this thing. The mixing of spaces and tabs alone makes the code embarrassingly difficult to read. http://imgur.com/a/XlSQD http://imgur.com/a/XlSQD

roblabla 9y ago
Fix your editor and set the tabspace to the correct value ? I'm guessing 2 spaces is the right value in this case.
- mapimopi 9y ago
  Or maybe it's the developer who needs to fix his editor. Proper editor should convert tabs into spaces or vise versa and not create complications for everyone else.
- mwpmaybe 9y ago
  Last I checked I don't have access to "fix" how GitHub displays source code, although it looks like the maintainers can add an .editorconfig to their project to address this particular case.

ccommsxx 9y ago

We're getting even better numbers on other devices: Android phone, ~5M ops/sec. Macbook Air, Chrome, ~30M ops/sec. Macbook Pro, Chrome Canary, ~80M ops/sec. Lenovo netbook, IE6, ~100K ops/sec [...] These numbers represent a breakthrough in performance not possible with other databases. Care to share some details on this benchmark? What kind of operations were these? Your results don't really sound likely for any kind of non-trivial operation - is it possible that you were testing a routine that was optimized out (to a static return/noop) by the JIT? from https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-Atom-CPU https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2...

benwilber0 9y ago
yeah 5 million "ops"/second on an Android phone? I want to know what the "ops" were. Is it just reading a value from a static memory address? Network request? What's going on here?
marknadal 9y ago
JS performance is terrible, it took us a year to rewrite everything from scratch to get these type of numbers. As mentioned near the bottom of that link, they are on cached reads which is unrealistic in real world settings but a useful baseline number to compare against. To understand our methodology on the benchmark approach, check out this tech talk on all the problems we encountered while testing (including things like you mentioned, dealing with how the JIT optimizes, or often times in Chrome, doesn't): https://youtu.be/BEqH-oZ4UXI https://youtu.be/BEqH-oZ4UXI However, a perk of the push-based model (realtime data sync like with Firebase), is that cached reads become a lot more realistic because often the data will already be there before you read it (unlike with a pull/poll based model). Any details I can expand on for you?
- benwilber0 9y ago
  > Android phone, ~5M ops/sec. what are the "ops" in this case?
  - 9y ago

akanet 9y ago

Do you plan to add presence to GUN? One of the key features that we require from Firebase is the ability to tell the server to delete a key in the event the client disconnects. https://firebase.google.com/docs/reference/android/com/google/firebase/database/OnDisconnect https://firebase.google.com/docs/reference/android/com/googl...

marknadal 9y ago
Not yet! Good suggestion though. Deletes right now are currently very difficult in gun. But we plan on fixing that in v0.8.x releases. Your idea of being able to do it on disconnects is a great one, I'll add it to the to-do list!

lacampbell 9y ago

This looks really cool. I'm no database expert, but a decentralised graph database is an intriguing idea. Has it been tried before? (decentralised databases I mean, not graph databases)

grandalf 9y ago
I'm building one, slightly different use case though.
- marknadal 9y ago
  Nice! Have any links to share?
  - grandalf 9y ago
    not yet but perhaps in the future. We're hiring fwiw.
jhoechtl 9y ago
https://dgraph.io/ https://dgraph.io/
vpol 9y ago
I hope it was a sarcastic comment.

maxdemarzi 9y ago

The docs seem to indicate you do not store properties on relationships? Is that right? Then why call it a graph database when it is really an RDF store?

marknadal 9y ago
You can do this, but it is not a built in primitive (like it is in Neo4j). We chose to do this because the majority of edges are usually directed, and we wanted to keep the base system as small as possible (all of gun is only about 12KB). It is easy to create "edge" nodes that store relationship properties, but nobody has created a full feature extension/framework for this yet, but I expect one will be built because this is a very useful feature to have.

deleted 9y ago

[deleted]

deleted 9y ago

[deleted]

jimktrains2 9y ago

The title and the first line of the README > GUN is a realtime, distributed, offline-first, graph database engine. Doing 15M+ ops/sec in just ~12KB gzipped. But no mention of any guarantees provided by the system. What kind of transactions are supported? are writes consensus-based, best effort, eventual consistent? The "GUN Survives a Primary Fault" raises more questions than answers. The "primary" appears to be the only server, and the clients are simply caching un-confirmed updates. Unless every client is a full copy of the database (which would seem prohibitive for the types of things this is aimed at?) there wasn't really any fault tolerance showed here. Also, all clients resolved to the most recent update, the one didn't fail because the row had been updated between the last read. While that's OK for some designs, it just smells fishy. I shouldn't be able to trash someone else's update. I feel like this is yet another datastore trying to call itself a database, but offering no guarentees of any kind that your data is safe.

marknadal 9y ago
The README mentions it in the #documentation section and links to these resources: (let me know how they can be improved) - CAP Theorem tradeoffs: https://github.com/amark/gun/wiki/CAP-Theorem https://github.com/amark/gun/wiki/CAP-Theorem (We are AP, which is eventually consistent) - My tech talk of how the CRDT works: http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html (compared to consensus protocols) Clients keep a copy of data that they are interested in (not the whole data set, correct), and together they can reconstruct the entire data set. You can also run multiple server peers that backup the whole data set, or use a shard key to determine which subsets. If this isn't fault tolerant, would you mind giving me some examples of what is? Yes, good point, to prevent data from getting trashed you need to run auth on the system, more info here: https://github.com/amark/gun/wiki/auth https://github.com/amark/gun/wiki/auth We certainly aren't a storage engine, but we are a database/datastore (whatever you want to call it), or more appropriately: an open data sync protocol. I have tried to make our guarantees/tradeoffs very well laid out in the above mentioned articles/README, anything I can do to make this more clear/obvious?
- jimktrains2 9y ago

anon12345690 9y ago

This poster has been working on this and spamming it all over HN for years, it's been proven to have major flaws and full of misleading claims with all those posts. millions of reads per second on mobile devices for a distributed database? just come on, why even post this?

marknadal 9y ago
Throughout the rest of the comments I've tried to provide a bunch of evidence/links for the performance claims (and tests that you can run yourself). Let me know if I missed anything and I'd love to chat about it! But proven flawed? Could you send me some links/evidence on that? I'd love to do my best to reply/address any concerns.