7 ms·
UnQLite – An Embeddable NoSQL Database Engine
- dfabulich 8y agoI think the FAQ needs another question: when should I prefer UnQLite over SQLite? When should I prefer SQLite over UnQLite?
- sephware 8y agoThat one seems straight forward: when you want SQL use SQLite, when you want NoSQL use UnQLite
- xellisx 8y agoFrom what I quickly gathered, it's just a Key-Value storage mechanism.
- sephware 8y agoThe title says NoSQL and the first paragraph on the page confirms that it's more than just a KV store: > UnQLite is a in-process software library which implements a self-contained, serverless, zero-configuration, transactional NoSQL database engine. UnQLite is a document store database similar to MongoDB, Redis, CouchDB etc. as well a standard Key/Value store similar to BerkeleyDB, LevelDB, etc.
- xellisx 8y agoAh, I didn't read it correctly then.
- deleted 8y ago[deleted]
- sitkack 8y agoChoices are often made across many dimensions.
- bvinc 8y agoThe question I'm more interested in is: what advantages does this provide vs creating my own document style schema in SQLite. Using the JSON1 extension to SQLite, you can even query inside of JSON blobs.
- dominotw 8y agoFrom reading the docs, I could spot the following differences 1. sqllite is 'sql' interface, unQlit is k/v. 2. Sqlite uses btree, unQlite is linear hash( doesn't support range queries).
- dfabulich 8y agoFrom the FAQ: "7. Is UnQLite thread-safe Threads are evil.[1] Avoid them. UnQLite is threadsafe and full re-entrant. But in order to be thread-safe, UnQLite must be compiled with the UNQLITE_ENABLE_THREADS compile time directive defined. If you are unsure if the UnQLite library you are linking against is compiled to be threadsafe you can call the unqlite_lib_is_threadsafe() interface to find out." [1] http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.... "The Problem with Threads" "Threads are evil" is not a useful remark to make in response to the question "is this library thread-safe"
- dominotw 8y agosqllite has the same warning in their FAQ's :D https://www.sqlite.org/faq.html#q6 https://www.sqlite.org/faq.html#q6
- blattimwind 8y agoSQLite can actually do grouping and sorting (iirc) in parallel, though it's disabled practically everywhere (Linux distros and other environments shipping SQLite seem to pick compile flags kinda arbitrarily, e.g. json1 and fts4/5 are fairly widespread on Linux, but not included in e.g. Python for Windows).
- otabdeveloper2 8y ago> An Embeddable NoSQL Database Engine So, std::unordered_map?
- bbcbasic 8y agoUnQLite reads and writes directly to ordinary disk files
- gmueckl 8y agoLooks more like an out of core unordered_map<string,string> with thread safety and persistence to me, if you really want to make that analogy.
- coleifer 8y agoNot really. I mean, yes, but it also has a concept of document storage and a php-like scripting language they call jx9.
- snissn 8y agothis is great - in my opinion there's a shortage of key value type data storage engines and there needs to be more competition in this regard. There simply aren't enough free/open source data engines to meet the demand of today's software. few quick thoughts - how does this compare to leveldb? How would it perform in something like geth or parity as an ethereum client back end data engine in compared to leveldb? do you have a python interface library?
- UncleEntity 8y agoFound a python2/3 bindings library on the github -- https://github.com/coleifer/unqlite-python https://github.com/coleifer/unqlite-python It's based on cython apparently, might dust off pybindgen and see if I can do better...
- coleifer 8y agoI'm the author of that library. I've worked to make sure all the unqlite features are available (transactions, cursors, jx9 vms, collections) and the interfaces are pythonic. Performance is good. Why not submit issues or patches instead?
- UncleEntity 8y ago> Why not submit issues or patches instead? Eh, not really a fan of cython... Been messing with pybindgen for a few hours now and will probably have the thing done by tomorrow, kind of want a drop in replacement (or as close as I can manage) for tinydb since I like their interface.
- manigandham 8y agoReally? There seem to be a ton of key/value engines on the market, especially because it's such a simple model to implement.
- dfox 8y agoThere is plethora of embedded K/V stores, but most of them only solve the simple to implement parts. That means either no concept of concurrency control or global RW-lock, zero resiliency to power failures, no support for atomic transactions... If you want on-disk K/V store for critical data you are pretty much stuck with BerkeleyDB.
- deleted 8y ago[deleted]
- interfixus 8y agoThe licensing is not altogether obvious. There seems to be a "one time fee" for downloading. The name sort of implies a connection with SQLite. No such connection exists. There is a parent company, name of "Symisc Systems". Which appears to be just one guy somewhere in France or Tunisia who likes to call himself "we". The code is not too kindly treated by several who have actually tried to use it. I think I'll pass.
- tyingq 8y agohttps://github.com/symisc/unqlite/blob/master/LICENSE https://github.com/symisc/unqlite/blob/master/LICENSE It does use a fair amount of sqlite source.
- MaxBarraclough 8y agoCharging for a 'production build' is certainly unusual. I'm reminded of when OpenBSD used to sell CDs.
- UncleEntity 8y ago> This production-ready package is available for a one-time fee of $20. Don't fault them for trying to make a bit of cash but going to have to pass since I'm too poor to spend $20 to see if it's a viable replacement for a little project I built on top of tinydb that's getting pretty slow since the db has grown past what a pure python db can comfortably handle. Or...I suppose they could provide a link from their download page to their github repo for us somewhat lazy folks -> https://github.com/symisc/unqlite https://github.com/symisc/unqlite
- evil-olive 8y agoIs there any documentation of the testing/verification process used in UnQLite? One of the things that gives me confidence in SQLite is https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html and it'd be interesting to see what approach UnQLite takes to the same problem.
- symisc_devel 8y agoHey all. I worked on the UnQLite development team back into 2013 ~ 2014. It was designed to be the primary storage for a home targeting router manufactured by a French tech conglomerate. The API is broken down into two independent layers plus one for various utility stuff like cross platform mmaped files, RC4 random numbers generation and so on. The first layer serve as a general purpose key/value store for the host application blobs (i.e binary or text data). You can store whole disk files inside a single database to generate a cross platform TAR archive for example. The second one is the document store layer similar in concept to what MongoDB offer but without the networking layer since everything run in the same process of the host application. Both layers are transactional and are able to recover after some external failure thanks to the SQLite journaling mechanism from which UnQLite is based on. Since its release in 2013, there was four major bugs including two critical that involves data loss under certain load. All known data corruption bugs were fixed in the last release and no major bugs were discovered a year or so. The library enjoy particular success among Python and C/C++ developers especially Chinese that used to bombard the original developers with various internal aspect of the library.
- ankrgyl 8y agoThis is a good example of someone twisting the term "serverless" to capture attention: > Serverless Database Engine > Most NoSQL database engines (i.e. MongoDB, Redis, CouchDB) are implemented as a separate server process. Programs that want to access the database communicate with the server using some kind of interprocess communication (typically TCP/IP) to send requests to the server and to receive back results. UnQLite like SQLite does not work this way. With UnQLite, the process that wants to access the database reads and writes directly from the database files on disk. There is no intermediary server process. The fact that it's embedded doesn't actually have anything to do with what popular "serverless" architectures are designed to solve, i.e. writing code that does not need to run in a specific server process to be invoked on demand. By their definition of serverless, pretty much any library is "serverless"... Edit: This is referring to "Classic Serverless" (https://sqlite.org/serverless.html https://sqlite.org/serverless.html) not "Neo-Serverless" architecture (my definition is more like "Neo-Serverless"). Thanks parhamn for pointing out the distinction.
- parhamn 8y agoThat doesn't seem fair. The word you quoted isn't even in the section you provided. The use of "server" is interchangeable with daemon/isolated-process here. That section seems like a decent explanation of daemon vs embedded for the junior dev.
- ankrgyl 8y agoHuh? It's literally in the "Serverless Database Engine" section. I can update the quote to include the header.
- parhamn 8y agoEven then, "twisting" is a stretch (especially the connoted maleficence). I think most people would realize these are unrelated and there seems to be a quite of precedence for talking about daemon-less as "serverless" [1]. [1] https://sqlite.org/serverless.html https://sqlite.org/serverless.html
- objektif 8y agoSo what are the advantages of this over say python dicts? Genuinely asking thanks.
- Buge 8y agoThe data in a python dict disappears when you turn your computer off. The data in UnQLite doesn't disappear.
- objektif 8y agoI could pickle the dict to a file right?
- Buge 8y agoIf you have 1GB of data, and want to change a small portion of it, that would require you to read the whole thing, then modify it, then write the whole thing back to disk. A database only requires a small amount of reading and writing.
- koenigdavidmj 8y agoSounds pretty similar in target usage to LMDB.
- newscracker 8y agoThe name though — what is it supposed to mean? “Un” as in “One” in French with QL Lite or “Un” as in a prefix for an opposite in English with QL Lite (like “No QL Lite”)?
- JulianMorrison 8y agoUn SQL = no sql And a pun on SQLite.
- reelly 8y ago> a pun on SQLite. And a bad one at that. I normally don't chime in on HN, but this pun is a crime against humanity.
- go_prodev 8y agoNot sure, but my first thought was Un(structured) Query Language
- teddyh 8y ago> UnQLite is a standard key/value store similar to BerkeleyDB, LevelDB, etc. but, with a rich feature set including support for transactions (ACID), concurrent reader, etc. This is slightly interesting, since the big reason I personally prefer applications using SQLite over those using BerkeleyDB (which used to be very common, btw) is that I’ve had lots of problems in the past with corrupted BerkeleyDB files. Like, almost all the time; if it used BerkeleyDB, it would sooner or later get corrupted. If UnQLite can be a “non-corrupting BerkeleyDB” for those who don’t actually need SQLite, it can serve an (admittedly niche) purpose.
- coleifer 8y agoOh it corrupts... https://github.com/symisc/unqlite/issues/41 https://github.com/symisc/unqlite/issues/41
- NonEUCitizen 8y agoIs there an "amalgamation file" like SQLite has for easy inclusion into other projects?
- rayascott 8y agoYes. unqlite.h/c in the root at https://github.com/symisc/unqlite https://github.com/symisc/unqlite
- coleifer 8y agoI authored a python binding and have some experience that may be relevant. The library seems poorly maintained in my opinion and might be abandoned. There have been critical bugs leading to data loss. They have a similar project vedis which is like an embedded redis that I think is completely abandoned. For the love of God just use SQLite if you want an embedded database for structured data, even if it's just eav. If you just need key/value there are battle-tested, well maintained options. https://github.com/symisc/unqlite/issues/41 https://github.com/symisc/unqlite/issues/41
- oscargrouch 8y ago> For the love of God just use SQLite if you want an embedded database for structured data, even if it's just eav. If you just need key/value there are battle-tested, well maintained options. You can also use the SQLite Disk Btree as a key-value data store. I've just implemented a solution that does so in C++ with a LevelDB-like api. You should use the backend Btree api (the same the VDBE/SQL engine uses to store stuff) and create 'tables' the SQLite uses for indexes. The integer-key based is used for ordinary SQL table .. (1 PK = 1 Record) and the blob-key type, where its agnostic about whats on the key. (Thats the one you use to create the KV store) There are a couple of details, to make it right, but its not that hard, and its a great solution.
- hyc_symas 8y agoThe Btree engine in SQLite is quite inefficient, and also unreliable on its own. (Needs the transaction log support to provide reliability.) LMDB blows it away. https://github.com/LMDB/sqlightning https://github.com/LMDB/sqlightning
- cwyers 8y agoYou've posted the same two-year-old closed issue twice on this thread. You're coming off as someone with an axe to grind.
- sitkack 8y agoPeople often get passionate about data corruption.
- dicroce 8y agoUnqlite doesn't support in order traversal of data because the default (and only included storage engine) is based on hashing.
- benchplz 8y agoshat code
- mankash666 8y agoThe main use case for NoSql is scale at the cost of consistency. What's the exact use case for UnQLite - as in - what scaling issues do embedded databases face? In what scenarios?