3 ms·

We're getting even better numbers on other devices: Android phone, ~5M ops/sec. Macbook Air, Chrome, ~30M ops/sec. Macbook Pro, Chrome Canary, ~80M op

We're getting even better numbers on other devices: Android phone, ~5M ops/sec. Macbook Air, Chrome, ~30M ops/sec. Macbook Pro, Chrome Canary, ~80M ops/sec. Lenovo netbook, IE6, ~100K ops/sec [...] These numbers represent a breakthrough in performance not possible with other databases. Care to share some details on this benchmark? What kind of operations were these? Your results don't really sound likely for any kind of non-trivial operation - is it possible that you were testing a routine that was optimized out (to a static return/noop) by the JIT? from https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-Atom-CPU https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2...

benwilber0 9y ago
yeah 5 million "ops"/second on an Android phone? I want to know what the "ops" were. Is it just reading a value from a static memory address? Network request? What's going on here?
marknadal 9y ago
JS performance is terrible, it took us a year to rewrite everything from scratch to get these type of numbers. As mentioned near the bottom of that link, they are on cached reads which is unrealistic in real world settings but a useful baseline number to compare against. To understand our methodology on the benchmark approach, check out this tech talk on all the problems we encountered while testing (including things like you mentioned, dealing with how the JIT optimizes, or often times in Chrome, doesn't): https://youtu.be/BEqH-oZ4UXI https://youtu.be/BEqH-oZ4UXI However, a perk of the push-based model (realtime data sync like with Firebase), is that cached reads become a lot more realistic because often the data will already be there before you read it (unlike with a pull/poll based model). Any details I can expand on for you?
- benwilber0 9y ago
  > Android phone, ~5M ops/sec. what are the "ops" in this case?
  - marknadal 9y ago
    Sorry, I was trying to reply to both of you but I was not clear - I apologize. The ops are cached reads also, it was the same test across different devices. For people wanting to run them yourself, clone the repo and go to test/ptsd/ptsd.html - we need to make it easier in the future though.
    - ccommsxx 9y ago
      Look not to sound harsh but since you're claiming the "80mm ops/sec" as a technical achievement "not possible with other databases" I found it to be fair to actually review the benchmark you linked. What you're benchmarking for the "read" is this: benchmark(function(){ gun.val(ok); }); Looking at what the gun.val call does it seems to be more or less an identitiy/noop function (that simply returns its input). It appears your benchmark is basically testing how fast you can do a simple javascript function call which has nothing to do with any database operation whatsoever? How is that useful? The benchmark doesn't reflect the performance of your gun database more than it reflects the performance of _any javascript application_. The only thing that is being benchmarked here is the javascript interpreter. Seriously, this is like somebody claiming their database runs at 4 billions ops/sec because that is how many instructions the CPU can perform. Given how much effort you (clearly) put into your marketing and the way you worded this as a "cached read" I think you probably know all of this though -- so here's a plea: Please don't pull these kinds of completely dishonest stunts (you even put the BS numbers in the title of this submission). You just make others in the javascript [database] space look bad by association. Cheating on benchmarks will not convince anybody but the most junior web developers. source (hopefully I'm wrong ;) ): https://github.com/amark/gun/blob/master/test/ptsd/perf.js#L1999 https://github.com/amark/gun/blob/master/test/ptsd/perf.js#L...
      - marknadal 9y ago
        The nitty gritty details are on this readthesource.io podcast: https://youtu.be/70dn1oZQFCk https://youtu.be/70dn1oZQFCk (watch on 2X speed). Each process does concurrency control and then has a centralized in-memory cache for the values (like what a lot of other in-memory databases do). So when `gun.val(cb)` is called, prototype context holds its value and is able to do an immediate read - without this, JS is so slow that every function call logarithmically decreases your performance (see the previously linked https://youtu.be/BEqH-oZ4UXI https://youtu.be/BEqH-oZ4UXI ). That is the advantage of the realtime/push-based model, you can cache most data before it is even read. Even in memory databases, like Redis and others, do this so there is no BS here, but I appreciate you trying to call it out. Additionally, not all in-memory reads are equal - my previous implementation of gun could only do a thousand or so reads/sec despite being cached. And please compare against other in-memory javascript database benchmarks: https://github.com/techfort/LokiJS/wiki/Indexing-and-Query-Performance https://github.com/techfort/LokiJS/wiki/Indexing-and-Query-P... (Joe's work is very good!) I sincerely wish it was as easy as leaving it up to the JS interpreter ;) but I've unfortunately found JS to be very slow. :P Hit me up with any other hard questions or if I missed anything. And please keep on trying hard to call out database vendors - I agree, it is extremely important to keep us open and honest. :) Cheers!

benwilber0 9y ago

yeah 5 million "ops"/second on an Android phone? I want to know what the "ops" were. Is it just reading a value from a static memory address? Network request? What's going on here?

marknadal 9y ago

JS performance is terrible, it took us a year to rewrite everything from scratch to get these type of numbers. As mentioned near the bottom of that link, they are on cached reads which is unrealistic in real world settings but a useful baseline number to compare against. To understand our methodology on the benchmark approach, check out this tech talk on all the problems we encountered while testing (including things like you mentioned, dealing with how the JIT optimizes, or often times in Chrome, doesn't): https://youtu.be/BEqH-oZ4UXI https://youtu.be/BEqH-oZ4UXI However, a perk of the push-based model (realtime data sync like with Firebase), is that cached reads become a lot more realistic because often the data will already be there before you read it (unlike with a pull/poll based model). Any details I can expand on for you?

benwilber0 9y ago
> Android phone, ~5M ops/sec. what are the "ops" in this case?
- marknadal 9y ago
  Sorry, I was trying to reply to both of you but I was not clear - I apologize. The ops are cached reads also, it was the same test across different devices. For people wanting to run them yourself, clone the repo and go to test/ptsd/ptsd.html - we need to make it easier in the future though.
  - ccommsxx 9y ago
    Look not to sound harsh but since you're claiming the "80mm ops/sec" as a technical achievement "not possible with other databases" I found it to be fair to actually review the benchmark you linked. What you're benchmarking for the "read" is this: benchmark(function(){ gun.val(ok); }); Looking at what the gun.val call does it seems to be more or less an identitiy/noop function (that simply returns its input). It appears your benchmark is basically testing how fast you can do a simple javascript function call which has nothing to do with any database operation whatsoever? How is that useful? The benchmark doesn't reflect the performance of your gun database more than it reflects the performance of _any javascript application_. The only thing that is being benchmarked here is the javascript interpreter. Seriously, this is like somebody claiming their database runs at 4 billions ops/sec because that is how many instructions the CPU can perform. Given how much effort you (clearly) put into your marketing and the way you worded this as a "cached read" I think you probably know all of this though -- so here's a plea: Please don't pull these kinds of completely dishonest stunts (you even put the BS numbers in the title of this submission). You just make others in the javascript [database] space look bad by association. Cheating on benchmarks will not convince anybody but the most junior web developers. source (hopefully I'm wrong ;) ): https://github.com/amark/gun/blob/master/test/ptsd/perf.js#L1999 https://github.com/amark/gun/blob/master/test/ptsd/perf.js#L...
    - marknadal 9y ago
      The nitty gritty details are on this readthesource.io podcast: https://youtu.be/70dn1oZQFCk https://youtu.be/70dn1oZQFCk (watch on 2X speed). Each process does concurrency control and then has a centralized in-memory cache for the values (like what a lot of other in-memory databases do). So when `gun.val(cb)` is called, prototype context holds its value and is able to do an immediate read - without this, JS is so slow that every function call logarithmically decreases your performance (see the previously linked https://youtu.be/BEqH-oZ4UXI https://youtu.be/BEqH-oZ4UXI ). That is the advantage of the realtime/push-based model, you can cache most data before it is even read. Even in memory databases, like Redis and others, do this so there is no BS here, but I appreciate you trying to call it out. Additionally, not all in-memory reads are equal - my previous implementation of gun could only do a thousand or so reads/sec despite being cached. And please compare against other in-memory javascript database benchmarks: https://github.com/techfort/LokiJS/wiki/Indexing-and-Query-Performance https://github.com/techfort/LokiJS/wiki/Indexing-and-Query-P... (Joe's work is very good!) I sincerely wish it was as easy as leaving it up to the JS interpreter ;) but I've unfortunately found JS to be very slow. :P Hit me up with any other hard questions or if I missed anything. And please keep on trying hard to call out database vendors - I agree, it is extremely important to keep us open and honest. :) Cheers!