4 ms·

While "Performance over everything" is obviously hyperbole, I hope they really don't mean it and actually prioritise correctness: that it apparently does actual

While "Performance over everything" is obviously hyperbole, I hope they really don't mean it and actually prioritise correctness: that it apparently does actually store and retrieve data suggests that at some level, at least, they do.

ascendantlogic 10y ago
MongoDB all over again.
- PeCaN 10y ago
  Well, if you're building the next SnapChat, MongoDB is an entirely reasonable option and may even save you some coding time!
  - curried_haskell 10y ago
    Simply use MongoDB with its default settings, and you're good to go with the next Snap-whatever product you want to build.
vidarh 10y ago
There are plenty of use-cases where the risk of losing data to gain performance is perfectly acceptable. What is important is being clear about what trade-offs they make. As an example: Years ago I worked on a classifieds aggregation engine; we crawled hundreds of thousands of feeds, indexing tens of millions of listings. All the queues in our processing pipeline were in-memory only: if we lost a few items (or a million) it didn't matter; worst case we'd pick them up no later than 24 hours later - in the case of more severe loss we'd simply trigger re-indexing of the feeds likely to have been "in flight" at the time. Failures would affect freshness and cause us to lose out on some updates, but as long as they were not severe enough or too frequent nobody would even notice, because the inventory was large and fleeting. There are lots of cases like that where "performance over everything" can be a totally valid trade-off, because your data store is not the canonical source of information, doesn't need to be precise or comprehensive, and/or can be easily refreshed, for example. This does require you to know and understand your requirements and what the datastore in question guarantees (or not), of course. If you don't, then you better stick with a "safe" option.
- andrewaylett 10y ago
  That's true, but it's got to be a trade-off and a balance of risk, because "Performance over everything" makes no sense at all: the most performant thing to do is to store no data and return no data. And even with that trade-off, I'd argue that there's a level of correctness that should be non-negotiable. Missing data might be OK, but corrupting data on the way in is bad, and failing to process everything might be OK (eventual consistency and all that) but you should meet your published guarantees and you shouldn't corrupt your results. So I would argue for correctness over performance, with the caveat that "correct" has a technical meaning specific to the project.
marknadal 10y ago
(Full disclosure, I work at a database company) Their performance is pretty impressive, but you can get over 30X their speed (30M ops/sec) by caching - see https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-Atom-CPU https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2... . And this can be done without compromising the storage or integrity of the data. The Observable pattern lets you update caches while you perform writes so the data stays correct, reflecting what is on disk. If caches don't exist then you pull from disk (slow) and write to the caches.
- nathancahill 10y ago
  > Caching speed is nx database speed. Well, no shit?
- joeminichino 10y ago
  this is actually something we are playing with :) we have a couple of drafts in the pipe but you are welcome to bring some wisdom over to LokiJS - it's free for all to use and for all to contribute, even if it's just on a theoretical level. Cheers
  - marknadal 10y ago
    Nice, very excited to hear! Honestly the wisdom I have has just been playing around with benchmark.js on V8 and noticing the Chrome team doesn't do as they claim (although latest Chrome Canary we saw major improvements), while FireFox is a different beast. You've probably done about just as much of this as me. What we found was that every function call in javascript significantly decreased performance (even if it did nothing) at a quazi logarithmic rate. So on a theoretical level if you're able to reuse function results that are deterministic you can save a ton of overhead. However IDK how micro-optimizing you want to get as it can sacrifice code legibility. Would love to chat more if you have time, I'll see if I can find your email and send you a message. Mine is (no spaces): a q u i v a @ google's email provider.

ascendantlogic 10y ago

MongoDB all over again.

PeCaN 10y ago
Well, if you're building the next SnapChat, MongoDB is an entirely reasonable option and may even save you some coding time!
- curried_haskell 10y ago
  Simply use MongoDB with its default settings, and you're good to go with the next Snap-whatever product you want to build.

vidarh 10y ago

There are plenty of use-cases where the risk of losing data to gain performance is perfectly acceptable. What is important is being clear about what trade-offs they make. As an example: Years ago I worked on a classifieds aggregation engine; we crawled hundreds of thousands of feeds, indexing tens of millions of listings. All the queues in our processing pipeline were in-memory only: if we lost a few items (or a million) it didn't matter; worst case we'd pick them up no later than 24 hours later - in the case of more severe loss we'd simply trigger re-indexing of the feeds likely to have been "in flight" at the time. Failures would affect freshness and cause us to lose out on some updates, but as long as they were not severe enough or too frequent nobody would even notice, because the inventory was large and fleeting. There are lots of cases like that where "performance over everything" can be a totally valid trade-off, because your data store is not the canonical source of information, doesn't need to be precise or comprehensive, and/or can be easily refreshed, for example. This does require you to know and understand your requirements and what the datastore in question guarantees (or not), of course. If you don't, then you better stick with a "safe" option.

andrewaylett 10y ago
That's true, but it's got to be a trade-off and a balance of risk, because "Performance over everything" makes no sense at all: the most performant thing to do is to store no data and return no data. And even with that trade-off, I'd argue that there's a level of correctness that should be non-negotiable. Missing data might be OK, but corrupting data on the way in is bad, and failing to process everything might be OK (eventual consistency and all that) but you should meet your published guarantees and you shouldn't corrupt your results. So I would argue for correctness over performance, with the caveat that "correct" has a technical meaning specific to the project.

marknadal 10y ago

(Full disclosure, I work at a database company) Their performance is pretty impressive, but you can get over 30X their speed (30M ops/sec) by caching - see https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-Atom-CPU https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2... . And this can be done without compromising the storage or integrity of the data. The Observable pattern lets you update caches while you perform writes so the data stays correct, reflecting what is on disk. If caches don't exist then you pull from disk (slow) and write to the caches.

nathancahill 10y ago
> Caching speed is nx database speed. Well, no shit?
joeminichino 10y ago
this is actually something we are playing with :) we have a couple of drafts in the pipe but you are welcome to bring some wisdom over to LokiJS - it's free for all to use and for all to contribute, even if it's just on a theoretical level. Cheers
- marknadal 10y ago
  Nice, very excited to hear! Honestly the wisdom I have has just been playing around with benchmark.js on V8 and noticing the Chrome team doesn't do as they claim (although latest Chrome Canary we saw major improvements), while FireFox is a different beast. You've probably done about just as much of this as me. What we found was that every function call in javascript significantly decreased performance (even if it did nothing) at a quazi logarithmic rate. So on a theoretical level if you're able to reuse function results that are deterministic you can save a ton of overhead. However IDK how micro-optimizing you want to get as it can sacrifice code legibility. Would love to chat more if you have time, I'll see if I can find your email and send you a message. Mine is (no spaces): a q u i v a @ google's email provider.