10 ms·
Ask HN: What is the SQLite of nosql databases?
I am looking for a simple, file-based nosql database. So basically the sqlite of nosql databases.
- throwaway888abc 5y agoCheck LiteStore as bonus you have api included https://h3rald.com/litestore/ https://h3rald.com/litestore/ https://github.com/h3rald/litestore https://github.com/h3rald/litestore
- deleted 5y ago[deleted]
- quantumofalpha 5y agoIsn't the main feature of nosql supposed to be easy horizontal scalability, the exact opposite of storing everything in a single file? If you just need a r/w store for some jsons in a single file, why not sqlite? You can put arbitrary-length blobs into it. Some sql will be involved but you can hide it in a wrapper class tailored to your application with a few dozen lines of code or so.
- etaioinshrdlu 5y agoPractically, NoSQL also seems to just mean "Not-SQL" as stuff like Redis is often lumped in it, which is about the opposite of easy horizontal scalability.
- gary_0 5y ago> You can put arbitrary-length blobs into it. And it's quite suitable for this purpose: https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html
- Jare 5y ago> Isn't the main feature of nosql supposed to be easy horizontal scalability There's no strict definition of nosql so everyone can choose their own. My personal take (in broad terms) follows: No, that's not a feature of nosql. Nosql means not relational, which in turn means no guarantees about the relationship between two objects, i.e. no atomicity of access across multiple objects (for either read or write operations). A consequence of this lack of atomicity is that it's easy to store different objects in different places, thus opening up opportunities for horizontal scalability. Caveat: those opportunities can be taken away by other choices you make. If you decide to offer and enforce transactions, you are bringing atomicity back into the system, and thus making horizontal scalability hard again. Or you may decide you want a nosql-in-a-file.
- xiphias2 5y agoBerkerleyDB is an older, simpler one, LevelDB / RocksDB are more modern, maintained, better for SSD workload
- Rochus 5y agoBerkerleyDB is rather complex and has a dual license (AGPL and commercial).
- erk__ 5y agoProbably something like LMDB [0] or Tkrzw [1], though nosql is a bit more diverse in a way SQL is not so it is hard to give a clear answer. [0]: https://en.m.wikipedia.org/wiki/Lightning_Memory-Mapped_Database https://en.m.wikipedia.org/wiki/Lightning_Memory-Mapped_Data... [1]: https://dbmx.net/tkrzw/ https://dbmx.net/tkrzw/
- eqvinox 5y ago+1 for LMDB, ended up there researching simple local key-value stores for my own use. GDBM and BerkeleyDB are the "grey beard" references, but their 80's & 90's heritage shows.
- adamansky 5y agoTake a look on the ejdb2 https://ejdb.org https://ejdb.org
- Blackthorn 5y agotkrzw is basically the modern dbm / berkeleydb
- dvfjsdhgfv 5y agoMy guess is that although you basically describe BerkeleyDB you probably want Redis.
- maxk42 5y agoTake a look at UnQLite: https://unqlite.org/ https://unqlite.org/
- andrey_utkin 5y agoFilesystem. Linux has VFS cache which is very robust and efficient. Remember that it is typical for programs to read /etc/nsswitch.conf, /etc/resolve.conf and tons of others at startup time - the filesystem is the datasource in Unix tradition, so the machinery is very well optimized.
- ludocode 5y agoThe problem with this is that if your records/documents are small, you're wasting huge amounts of space because each file uses a full filesystem block. If you have, say, ten thousand records where each is 200 bytes, a decent database would store that in a bit over 2MB. Storing these as individual files on a filesystem with 4kB blocks will take up at least 40MB. This is a huge amount of wasted space, not to mention slow. (Some filesystems do support tail packing but that won't fully solve the problem.) Not to mention all the other problems with this. The filesystem has a complete lack of higher-level features: no transactions, no snapshots, no indexing beyond filenames, no easy robustness guarantees (doing fsync() properly is a lot more complicated than it appears.) Honestly for modern apps the filesystem is just terrible at storing any internal mutable app data. Once you start writing code to store auxiliary indices, synchronize writes, or pack multiple records per file, well at that point you're just implementing your own database. This might make sense if, say, you have a special way of compressing your data (like git). But generally you're better off using a real embedded database.
- inshadows 5y agoSome filesystem can store small data directly in inode. Limit for ext4 is 160 bytes though. https://unix.stackexchange.com/questions/197570/is-it-possible-to-store-data-directly-inside-an-inode-on-a-unix-linux-filesyst https://unix.stackexchange.com/questions/197570/is-it-possib... https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Inline_Data https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Inli...
- deleted 5y ago[deleted]
- creshal 5y agoThe SQLite of NoSQL is still SQLite: https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html
- ludocode 5y agoAs others have mentioned you have lots of options: LMDB, LevelDB/RocksDB, BerkeleyDB. For what it's worth, I spent a long time looking for an embedded key-value store for my current native project since I didn't need the full complexity of SQL. In the end I chose... SQLite. All of these embedded NoSQL databases seem to be missing critical features. One such feature for my use case is database compaction. Last I checked, an LMDB database file can never shrink. Full compaction of LevelDB is slow and complicated (as I understand it essentially breaks the levels optimization which is the whole point of the thing.) SQLite meanwhile supports fast incremental vacuum, and it can be triggered manually or automatically. SQLite just has everything. Plus the reliability is unmatched. Even if you just need a single table that maps blob keys to blob values, I would still recommend SQLite over any NoSQL database today.
- justsomeuser 5y agoAlso there are high quality libraries for every language
- amachefe 5y agoFor now, see Postgres as one
- pistoriusp 5y agoJson on the filesystem
- RustyRussell 5y agoI'd nominate TDB (the trivial database). It's small, robust, effective
- teekay 5y agoOne of those I've tried is LiteDB - https://github.com/mbdavid/LiteDB https://github.com/mbdavid/LiteDB. I liked it. It's small yet capable. If you are familiar with MongoDB, you will feel right at home. It's great for .NET developers as it's written in C# but since it's Netstandard 1.3 compatible, you can presumably run it under Ubuntu or Mac OS or wherever else the new .NET 5 runtime works. I've got a C# app running on ARM64 the other day - just saying. I wrote about my experience playing with LiteDB here - https://tomaskohl.com/code/2020-04-07/trying-out-litedb/ https://tomaskohl.com/code/2020-04-07/trying-out-litedb/. It's not an in-depth look at all, just a few notes from the field, so to speak.
- voodoochilo 5y agohttps://unqlite.org/ https://unqlite.org/
- kenOfYugen 5y agoI ended up writing a small wrapper on top of SQLite based on this: https://dgl.cx/2020/06/sqlite-json-support https://dgl.cx/2020/06/sqlite-json-support With proper concurrency control, it can work very well even for multi process applications.
- fulafel 5y agoNoSQL databases have many different different data models. Eg object, document, graph, and key/value DBs. In a lot of cases you should probably just use something on top of SQLite, but you should say more about your requirements. An interesting one I ran into recently is Datalevin, a Datalog DB on top of LMDB for Clojure: https://github.com/juji-io/datalevin https://github.com/juji-io/datalevin
- tincholio 5y agoI'd add Crux to this... it's document-oriented, but still generates attribute-level indices, and is also Datalog based. It can be used on top of RocksDB, LMDB, or many other (more scalable) backends (Kafka being the canonical one). It's really awesome, and the team behind it are super responsive and helpful.
- nicodds 5y agoPersonally, I like very much leveldb/rocksdb. They're very fast and solid.
- 1vuio0pswjnm7 5y agohttps://en.wikipedia.org/wiki/Cdb_(software) https://en.wikipedia.org/wiki/Cdb_(software)
- basiclaser 5y agoLodb
- Tpt 5y agoI like sled that is a nice embedded key value store written in Rust: https://sled.rs/ https://sled.rs/ However, it is still in heavy development and a bit of a moving target even if the developers are currently heading toward stabilization of the file format.
- Rochus 5y agoSQLite has a backend which is well suited as a key-value store. Here is a NoSql database based on the SQLite backend: https://github.com/rochus-keller/Udb https://github.com/rochus-keller/Udb. I use it in many of my apps, e.g. https://github.com/rochus-keller/CrossLine https://github.com/rochus-keller/CrossLine. It's lean and fast, and supports objects, indices, hierarchical "globals" like ANSI-M and transactions.
- Abishek_Muthian 5y agoThere are also persistent key-value store like dbm as part of standard library in python and several 3rd party implementations like bitcask for Go.
- quickthrower2 5y agoAn object reference :-) In memory, high performance, no schema. Get the object to journal to disk and you are almost there!
- abrookewood 5y agoI think you want Mongita. It was featured on HN a while ago: "Mongita is to MongoDB as SQLite is to SQL (github.com/scottrogowski)" https://news.ycombinator.com/item?id=26881915 https://news.ycombinator.com/item?id=26881915
- jf22 5y agoRavenDb would be what I would use.
- CRConrad 5y agoMostly, but actually not totally, kidding: *.INI files.
- diehunde 5y agoYou mean an embedded NoSQL database? Because all databases whether SQL or NoSQL are file-based with just different formats and structures.
- sureshdasari 5y agoIts still sqlite. You can get more info here https://www.tutlane.com/tutorial/sqlite https://www.tutlane.com/tutorial/sqlite
- tutlane 5y agoIts same as sqlite. You can get more info in this sqlite tutorial. https://www.tutlane.com/tutorial/sqlite https://www.tutlane.com/tutorial/sqlite
- pipework 5y agoHey sureshdasari, this is probably against the rules for advertising.