12 ms·

LokiJS – Lightweight JavaScript in-memory database

130 points by joeminichino 12y ago

leeber 12y ago
This looks interesting. Coincidentally I was just searching around for something like this to run in the browser. I'll play around with it myself, but is there anyone who has used this that knows how well it does with memory usage?
- nobodysfool 12y ago
  Uhm, I'm pretty sure it can use all your ram if you want. It's an in-memory database, so all your ram will be used for data...
  - RussianCow 12y ago
    I think the question was more about how compactly it stores the data in memory.
- yourad_io 12y ago
  +1 for interesting. The ToDo demo shows the size of the collection [1]. Seems rather efficient. [1] http://lokijs.org/#/demo http://lokijs.org/#/demo
Sir_Cmpwn 12y ago
Related: SQL.js, which is sqlite compiled with emscripten https://github.com/kripken/sql.js https://github.com/kripken/sql.js
- indubitably 12y ago
  Except that file is 10x bigger than loki…
  - bpicolo 12y ago
    So? It's also full-blown sqlite. Let's not pretend file sizes matter much in modern times.
    - rakoo 12y ago
      It does if you're going to send that to a browser.
jedireza 12y ago
I'm curious why NeDB wouldn't suffice. https://github.com/louischatriot/nedb https://github.com/louischatriot/nedb
- yourad_io 12y ago
  From the slideshare presentation on the homepage (p. 4)[1] > Performs better than similar products (NeDB, TaffyDB, PouchDB etc.) and it's much smaller It would be interesting to see benchmarks. [1] http://www.slideshare.net/techfort/lokijs/4 http://www.slideshare.net/techfort/lokijs/4
  - fiatjaf 12y ago
    Ok, TaffyDB is a piece of crap. You call `.fetch` to fetch a document at some id, but if you modify your fetched copy, the copy in the database is also modified. With a database like this I'm much better with a hashmap.
  - jedireza 12y ago
    Thanks for sharing that. I jumped the gun before commenting.
    - yourad_io 12y ago
      No worries. I just noticed they include the benchmarks here [1] [1] https://github.com/techfort/LokiJS/tree/master/benchmark https://github.com/techfort/LokiJS/tree/master/benchmark
  - kristopolous 12y ago
    there is this benchmark : https://github.com/danstocker/jorder/wiki/Benchmarks https://github.com/danstocker/jorder/wiki/Benchmarks ... I maintain this one (https://github.com/kristopolous/db.js https://github.com/kristopolous/db.js) which is part of that. From my perspective, with a small collection size, I'll sacrifice performance for expressiveness. I have a number of expensive features that I think do nice things.
remon 12y ago
Out of pure curiosity, what are the common use cases for a fast in-memory database? Are they exclusive to server-side applications (e.g. caching)?
- rmsaksida 12y ago
  Could be useful for a browser app which handles 'large' datasets, or for a mobile/desktop app as an SQLite replacement. Think of a messaging app which lets you search conversations client side with decent performance. It has a small footprint and some interesting features, so it might be a convenient DB for some apps even without strong performance requirements.
- ilghiro 12y ago
  Cordova is a big one. The ability to have rich queries into some persistent data set is something that's difficult to achieve atm.
  - joeminichino 12y ago
    It's precisely why I created LokiJS in the first place, then it grew much bigger than that but yeah - that is where it all came from.
    - nbrogi 12y ago
      Bravo, Joe. Good job.
- delluminatus 12y ago
  You can also use them in the browser to cache data to reduce server calls or improve client UI performance (a la meteor.js).
- tjelen 12y ago
  Well, it could be very useful as a more advanced "model" layer in browser-side MVC apps. I'm currently working on something that has a pretty complex data structure and simple JS objects/arrays simply don't cut it. LokiJS certainly seems worth looking into. The feature that looks especially promising is the ability to create Dynamic Views and to listen on notifications on these views.
skrebbel 12y ago
Great! I'm really no expert, but I think that language-specific persistent in-memory databases are the way to go. Deep inside, what I really want on my backends is just a data structure. A single big tree (or graph) of objects. Some primitives to make it thread-safe somehow (immutable collections, or mutexes, or just old-fashioned single-threadedness like Node - whatever), and that's it. Requests from the client query and mutate that data. In very many situations, I wouldn't need anything special to make this work fine with an application up to a pretty large number of users. Not all apps do big data analyses. RAMs are gigantic these days, bigger than the SQL datafiles of swathes of webapps. However, there's that pesky problem of code upgrades and crashing servers. If my server would never crash and I could hot-swap the code, I wouldn't need persistence at all. But I do, so I don't just want an in-memory datastructure, but a persistent in-memory data structure. I want to use this datastructure as easily as the programming language could possibly make it be for me while still guaranteeing some sane level of persistence and fault tolerance. Redis is a nice idea, but the fact that it's a separate server, with a client and a protocol and data structures that just not precisely map to my programming language's data structures force me to write a whole bunch of boilerplate anyway. Even more so with other databases like Mongo or Postgres. If I understand it well, Loki doesn't entirely do what I'd want: it does not save the data to disk on every change, but less often, if I get it right. That might be good enough if my problem allows for many little independent Loki databases, but if the dataset is a gigabyte and persistence means flushing that whole gigabyte to disk after every data change, it probably won't work very well. I'm really curious if other people here have similar ideas, maybe implementations, of these concepts. Maybe Loki could be extended with a snapshot+operation_log type of data storage like Redis has and then it'd be pretty close.
- yourad_io 12y ago
  > If I understand it well, Loki doesn't entirely do what I'd want: it does not save the data to disk on every change, but less often, if I get it right. From a cursory glance at the source, it doesn't look like anything internal ever calls .save()/.saveToDisk(), so it is up to you to decide how often you want to be creating your on-disk checkpoints. EDIT: Seems I am wrong: "In a nodejs/node-webkit environment LokiJS also persists to disk whenever an insert, update or remove is performed." [1]. But I still can't spot where .save[/ToDisk] is called in the source [2]. Anyone? > That might be good enough if my problem allows for many little independent Loki databases, but if the dataset is a gigabyte and persistence means flushing that whole gigabyte to disk after every data change, it probably won't work very well. If you're looking at gigabyte+ datasets, then something designed to be in-memory is probably not your best bet. Aside from saving, (one imagines) this would also affect load times (reading->parsing->importing a multi-GB JSON file at launch can't be quick). From [1]: > LokiJS is ideal for the following scenarios: * where a lightweight in-memory db is ideal * cross-platform mobile apps where you can leverage the power of javascript and avoid interacting with native databases * data sets are not so large that it wouldn't be a problem loading the entire db from a server and synchronising at the end of the work session [1] https://github.com/techfort/LokiJS https://github.com/techfort/LokiJS [2] https://github.com/techfort/LokiJS/blob/master/src/lokijs.js https://github.com/techfort/LokiJS/blob/master/src/lokijs.js
keithwhor 12y ago
How does this compare to DataCollection.js? https://www.npmjs.org/package/data-collection https://www.npmjs.org/package/data-collection http://thestorefront.github.io/DataCollection.js/ http://thestorefront.github.io/DataCollection.js/ They look fairly similar. What's the test coverage and do you have performance benchmarks?
cloudsheet 12y ago
Has anyone used this is lieu of or in comparison to Redis? Am I correct in assuming that with LokiJS, one could move in-memory data storage from the server (with Redis) to the browser (with LokiJS)? This project looks quite interesting. Thanks for sharing. Could be a great solution for single-page web apps (and mobile, Cordova, etc.).
- panzi 12y ago
  If you want a simple key-value-store in the browser, why don't you directly use localStorage(+JSON)? It's supported since IE8. My guess is that this thing is using localStorage under the hood.
  - RussianCow 12y ago
    Presumably because LokiJS is faster. (I don't know if that's actually true.) And no, I just looked at the source code and Loki does not use localStorage under the hood. It's an in-memory database without a persistence layer, so you lose the data when you leave the page (in the browser). Edit: Actually, reading more about Loki, it looks like the whole point of it is to provide querying/indexing capabilities to data in memory. So I don't know if there would be any value in using it as a replacement for key/value stores.
imslavko 12y ago
From skimming the presentation it looks like a project similar to Meteor's Minimongo and Miniredis (https://www.meteor.com/mini-databases https://www.meteor.com/mini-databases) used for browser caches. Minimongo, for example, implements a great majority of Mongo selectors (there are no secondary indexes, though). (I contributed to both Minimongo and Miniredis)
quartzmo 12y ago
I want a very fast, client-side, read-only database with rich query support. Is this the best choice?
marknadal 12y ago
Excellent project, there needs to be more focus in this space. I've been feeling a general trend of developers going back to basic embedded databases. I'm working on a similar project, that is focused on ease of use combined with distributed/decentralized (concurrency safe) behavior. Http://github.com/amark/gun

130 points by joeminichino 12y ago

This looks interesting. Coincidentally I was just searching around for something like this to run in the browser. I'll play around with it myself, but is there anyone who has used this that knows how well it does with memory usage?

nobodysfool 12y ago
Uhm, I'm pretty sure it can use all your ram if you want. It's an in-memory database, so all your ram will be used for data...
- RussianCow 12y ago
  I think the question was more about how compactly it stores the data in memory.
yourad_io 12y ago
+1 for interesting. The ToDo demo shows the size of the collection [1]. Seems rather efficient. [1] http://lokijs.org/#/demo http://lokijs.org/#/demo

Sir_Cmpwn 12y ago

Related: SQL.js, which is sqlite compiled with emscripten https://github.com/kripken/sql.js https://github.com/kripken/sql.js

indubitably 12y ago
Except that file is 10x bigger than loki…
- bpicolo 12y ago
  So? It's also full-blown sqlite. Let's not pretend file sizes matter much in modern times.
  - rakoo 12y ago
    It does if you're going to send that to a browser.

jedireza 12y ago

I'm curious why NeDB wouldn't suffice. https://github.com/louischatriot/nedb https://github.com/louischatriot/nedb

yourad_io 12y ago
From the slideshare presentation on the homepage (p. 4)[1] > Performs better than similar products (NeDB, TaffyDB, PouchDB etc.) and it's much smaller It would be interesting to see benchmarks. [1] http://www.slideshare.net/techfort/lokijs/4 http://www.slideshare.net/techfort/lokijs/4
- fiatjaf 12y ago
  Ok, TaffyDB is a piece of crap. You call `.fetch` to fetch a document at some id, but if you modify your fetched copy, the copy in the database is also modified. With a database like this I'm much better with a hashmap.
- jedireza 12y ago
  Thanks for sharing that. I jumped the gun before commenting.
  - yourad_io 12y ago
    No worries. I just noticed they include the benchmarks here [1] [1] https://github.com/techfort/LokiJS/tree/master/benchmark https://github.com/techfort/LokiJS/tree/master/benchmark
- kristopolous 12y ago
  there is this benchmark : https://github.com/danstocker/jorder/wiki/Benchmarks https://github.com/danstocker/jorder/wiki/Benchmarks ... I maintain this one (https://github.com/kristopolous/db.js https://github.com/kristopolous/db.js) which is part of that. From my perspective, with a small collection size, I'll sacrifice performance for expressiveness. I have a number of expensive features that I think do nice things.

remon 12y ago

Out of pure curiosity, what are the common use cases for a fast in-memory database? Are they exclusive to server-side applications (e.g. caching)?

rmsaksida 12y ago
Could be useful for a browser app which handles 'large' datasets, or for a mobile/desktop app as an SQLite replacement. Think of a messaging app which lets you search conversations client side with decent performance. It has a small footprint and some interesting features, so it might be a convenient DB for some apps even without strong performance requirements.
ilghiro 12y ago
Cordova is a big one. The ability to have rich queries into some persistent data set is something that's difficult to achieve atm.
- joeminichino 12y ago
  It's precisely why I created LokiJS in the first place, then it grew much bigger than that but yeah - that is where it all came from.
  - nbrogi 12y ago
    Bravo, Joe. Good job.
delluminatus 12y ago
You can also use them in the browser to cache data to reduce server calls or improve client UI performance (a la meteor.js).
tjelen 12y ago
Well, it could be very useful as a more advanced "model" layer in browser-side MVC apps. I'm currently working on something that has a pretty complex data structure and simple JS objects/arrays simply don't cut it. LokiJS certainly seems worth looking into. The feature that looks especially promising is the ability to create Dynamic Views and to listen on notifications on these views.

skrebbel 12y ago

Great! I'm really no expert, but I think that language-specific persistent in-memory databases are the way to go. Deep inside, what I really want on my backends is just a data structure. A single big tree (or graph) of objects. Some primitives to make it thread-safe somehow (immutable collections, or mutexes, or just old-fashioned single-threadedness like Node - whatever), and that's it. Requests from the client query and mutate that data. In very many situations, I wouldn't need anything special to make this work fine with an application up to a pretty large number of users. Not all apps do big data analyses. RAMs are gigantic these days, bigger than the SQL datafiles of swathes of webapps. However, there's that pesky problem of code upgrades and crashing servers. If my server would never crash and I could hot-swap the code, I wouldn't need persistence at all. But I do, so I don't just want an in-memory datastructure, but a persistent in-memory data structure. I want to use this datastructure as easily as the programming language could possibly make it be for me while still guaranteeing some sane level of persistence and fault tolerance. Redis is a nice idea, but the fact that it's a separate server, with a client and a protocol and data structures that just not precisely map to my programming language's data structures force me to write a whole bunch of boilerplate anyway. Even more so with other databases like Mongo or Postgres. If I understand it well, Loki doesn't entirely do what I'd want: it does not save the data to disk on every change, but less often, if I get it right. That might be good enough if my problem allows for many little independent Loki databases, but if the dataset is a gigabyte and persistence means flushing that whole gigabyte to disk after every data change, it probably won't work very well. I'm really curious if other people here have similar ideas, maybe implementations, of these concepts. Maybe Loki could be extended with a snapshot+operation_log type of data storage like Redis has and then it'd be pretty close.

yourad_io 12y ago
> If I understand it well, Loki doesn't entirely do what I'd want: it does not save the data to disk on every change, but less often, if I get it right. From a cursory glance at the source, it doesn't look like anything internal ever calls .save()/.saveToDisk(), so it is up to you to decide how often you want to be creating your on-disk checkpoints. EDIT: Seems I am wrong: "In a nodejs/node-webkit environment LokiJS also persists to disk whenever an insert, update or remove is performed." [1]. But I still can't spot where .save[/ToDisk] is called in the source [2]. Anyone? > That might be good enough if my problem allows for many little independent Loki databases, but if the dataset is a gigabyte and persistence means flushing that whole gigabyte to disk after every data change, it probably won't work very well. If you're looking at gigabyte+ datasets, then something designed to be in-memory is probably not your best bet. Aside from saving, (one imagines) this would also affect load times (reading->parsing->importing a multi-GB JSON file at launch can't be quick). From [1]: > LokiJS is ideal for the following scenarios: * where a lightweight in-memory db is ideal * cross-platform mobile apps where you can leverage the power of javascript and avoid interacting with native databases * data sets are not so large that it wouldn't be a problem loading the entire db from a server and synchronising at the end of the work session [1] https://github.com/techfort/LokiJS https://github.com/techfort/LokiJS [2] https://github.com/techfort/LokiJS/blob/master/src/lokijs.js https://github.com/techfort/LokiJS/blob/master/src/lokijs.js

keithwhor 12y ago

How does this compare to DataCollection.js? https://www.npmjs.org/package/data-collection https://www.npmjs.org/package/data-collection http://thestorefront.github.io/DataCollection.js/ http://thestorefront.github.io/DataCollection.js/ They look fairly similar. What's the test coverage and do you have performance benchmarks?

cloudsheet 12y ago

Has anyone used this is lieu of or in comparison to Redis? Am I correct in assuming that with LokiJS, one could move in-memory data storage from the server (with Redis) to the browser (with LokiJS)? This project looks quite interesting. Thanks for sharing. Could be a great solution for single-page web apps (and mobile, Cordova, etc.).

panzi 12y ago
If you want a simple key-value-store in the browser, why don't you directly use localStorage(+JSON)? It's supported since IE8. My guess is that this thing is using localStorage under the hood.
- RussianCow 12y ago
  Presumably because LokiJS is faster. (I don't know if that's actually true.) And no, I just looked at the source code and Loki does not use localStorage under the hood. It's an in-memory database without a persistence layer, so you lose the data when you leave the page (in the browser). Edit: Actually, reading more about Loki, it looks like the whole point of it is to provide querying/indexing capabilities to data in memory. So I don't know if there would be any value in using it as a replacement for key/value stores.

imslavko 12y ago

From skimming the presentation it looks like a project similar to Meteor's Minimongo and Miniredis (https://www.meteor.com/mini-databases https://www.meteor.com/mini-databases) used for browser caches. Minimongo, for example, implements a great majority of Mongo selectors (there are no secondary indexes, though). (I contributed to both Minimongo and Miniredis)

quartzmo 12y ago

I want a very fast, client-side, read-only database with rich query support. Is this the best choice?

marknadal 12y ago

Excellent project, there needs to be more focus in this space. I've been feeling a general trend of developers going back to basic embedded databases. I'm working on a similar project, that is focused on ease of use combined with distributed/decentralized (concurrency safe) behavior. Http://github.com/amark/gun