8 ms·
Sophia – An embeddable key-value database
- clumsysmurf 13y agoFor Java / Android, I've been using H2's MVStore, which is log structured and uses counted B+-trees. It's nice not having to go through JNI for good performance in Java. http://www.h2database.com/html/mvstore.html http://www.h2database.com/html/mvstore.html
- eropple 13y agoFunny - H2's slowness (either with a standard storage system or with MVStore and a standard key format) is the main reason we're moving back to a hand-rolled data storage system that's specific for our data on Android.
- clumsysmurf 13y agoThomas Mueller, the author of H2 / MVStore, gives some thoughts on H2's performance issues under Android here if you are interested: https://groups.google.com/forum/#!topic/h2-database/Q8K-nbChf1w https://groups.google.com/forum/#!topic/h2-database/Q8K-nbCh...
- eropple 13y agoYeah, I saw that. The main problem for us is that we need on-disk encryption due to regulatory issues and the encrypted SQLCipher build was causing us no end of grief. And we don't really need a SQL database, it's just what the developers of our iOS app were doing and--it being our first Android mobile app--we thought it was a good idea to do the same. V2 is ripping that out both for perf and for code-sanity reasons.
- scanr 13y agoThat looks awesome. Thanks for pointing it out.
- Goopplesoft 13y agoVery cool. As a suggestion, increase the link size under the main title, wasn't clear to me what the next step was at first after reading the introduction text.
- conductor 13y agoI love the simplicity of the site and the C code. I will definitely use it, thank you.
- dfischer 13y agoTypography is hard to read.
- rgbrgb 13y agoI know it seems superficial but beautiful docs are one of my most trusted heuristics when I'm considering using a library. If the author cares about the aesthetics of the docs, it often means they care about the aesthetics of the code, which really does matter a lot. We can make ugly things or we can make beautiful things. I really respect people who take the time to make beautiful tools.
- VikingCoder 13y agoWas that intended to be a compliment of this project? I had the opposite reaction. I accept that English is often a second language, but this page was off-putting. Sophia is a modern [should add comma] embeddable key-value database designed for a high [should hyphenate] load environment. It has a unique architecture that was born as a result of research and rethinking primary alghorithmical [sic, should say "algorithm"] constraints associated with a [sic] getting popular Log-file based data structures, such as LSM-tree [should say "trees"], it's [sic] variations based on Fractional Cascading ideas and a B-Tree. (see architecture) [run-on, meaning unclear] It is very fast. (see benchmarks) it [sic] is easy to use. (see documentation) Implemented as a small C [should probably hyphenate, or just rewrite] written, BSD [should probably hyphenate] licensed library.
- rgbrgb 13y agoThey should definitely do some editing. The grammatical errors are kind of off putting, I totally agree. That said there's a very clear value prop/use case and clear, concise documentation. You may be getting downvoted because your language was rather harsh.
- dilap 13y agoI don't think you're weighing the "English-as-a-second" language factor nearly strongly enough. While grammar can be a great proxy measure of quality for works by native speakers, it's completely inappropriate for non-native speakers -- the amount of effort required to reach native-level fluency and polish (i.e., at the level of your critique, above) is herculean. To cite just one example, redis is widely considered excellent software, but much of its documentation is written in a very imperfect English.
- hosay123 13y agoLooks nice, but note this doesn't appear to support consistent reads (unlike LevelDB snapshots)
- maaku 13y agoSnapshots? I could find it in the documentation..
- luisbebop 13y agoAwesome work, congratulations!
- saurik 13y agoAnyone know anything about how this compares in practice to Lightning MDB (which uses a memory mapped B-tree, I think, and is apparently insanely faster than most of the other enbedded key-value stores people normally examine)?
- HNLogInShit 13y agoThe term is "key/value". http://goldmanosi.blogspot.com/2012/04/when-did-slash-become-too-hard.html http://goldmanosi.blogspot.com/2012/04/when-did-slash-become...
- hosay123 13y agoThis one copies, and it has no concept of transactions from the looks of it (not even LevelDB-style snapshots)
- MichaelGG 13y agoI've got a need for something like this, but would like to have the keys and values delta encoded to achieve simple, yet effective, compression.
- leif 13y agoHow about http://github.com/Tokutek/ft-index http://github.com/Tokutek/ft-index? It's embeddable (BDB-like API), has compression built in, and is a similar data structure to this but with more mature features like transactions.
- jgalt212 13y agoany word on support for unicode keys?
- dlundqvist 13y agoKeys are arbitrary data (you pass in pointer to data and length in bytes), so you can use anything that makes sense for you as keys.
- apendleton 13y agoHaving just gone through the exercise of picking an embedded key-value store for a project, some things that would be nice: how does it compare to other things besides leveldb (which, to be frank, isn't a stellar performer)? In particular, how does it compare to Tokyo/Kyoto Cabinet, Lightning MDB, or Sqlite4's LSM? Does it support data compression (either with a single pre-selected algorithm like LevelDB and Snappy, or in pluggable fashion like LSM)? How does it deal with concurrent access by multiple processes?
- shepik 13y agoI've gone through picking embedded key-value store, too. What really concerns me is why never in benchmarks they perform on already filled database (like, 14G, 28G, 60G)? Because "add 100k random keys into an empty database" is very different from "add 100k random keys into a large database". And that is where more novel algorythms start to shine. Yes, read speed of leveldb (and, i assume, sophia) with its fancy sst's is lower than of plain old b-trees or hashtables (kctree/kchash), but it is still high enough for most tasks. Write performance of kc* (and btree-based libraries in general) is, however, unacceptable, at least on hard drives, and even with a reasonable-sized database (~90% of RAM) it degrades to a couple of random write per IOPs (so, 200-300 writes per second on a consumer-grade HDD, or up to 1000 on a 2x10k sas hdd in raid-0, if i remember correctly) It may be reasonable to use kc* on SSD, but i did not test that.
- Negitivefrags 13y agoIt says that the benchmark source is on github, but I can't find it. It doesn't appear to be in their primary repo. I would like to try and do my own test against another embedded data store like Berkeley DB but I want to know more about the conditions on the test. How many threads were used, that kind of thing.
- the1 13y agohttps://github.com/pmwkaa/sophia_benchmark https://github.com/pmwkaa/sophia_benchmark
- laichzeit0 13y agoIn case any of the devs read this: 1. Can multiple processes use the same database concurrently? (Separate address space processes, not fork()'d) 2. Have you tested this with uClib/cross compiler? (I would like to use it on a MIPs embedded router) The reason I ask this is because I recently had the displeasure of having to hack a non-volatile RAM library to work with shared memory / thread safe and something small like this would be a perfect replacement with a lot less pain.
- acron0 13y agoHad a quick, 30 min bash at a win32 port using msinttypes and pthread-win32 but no luck yet :( Would love to see one though...
- pwpwp 13y agoI, for one, wouldn't trust my data to a library by somebody who uses the same text decoration for hyperlinks as for plain text.
- buster 13y agoI'd be far more interested in benchmarks versus BDB (and maybe even sqlite).
- ksec 13y agoSomething for Mozilla to consider using inside Firefox inplace of LevelDB ( If that was ever landed )
- dkhenry 13y agoWas this man's computer use being charged by the key stroke? I mean I understand using a few abbreviations here and there, but i.c ? At least name your files descriptively. I would avoid using this for realzies if only for the fact that if something broke trying to fix it in that code base would be prohibitive
- i_have_to_speak 13y agoCute website. Some random thoughts: Concurrency: - No mention of it. There appear to be spin locks in the source. No multi-threaded tests. Stability and data safety: - Github has 2 days of history, and 4kLoC of test code. Why should I trust my data to you? "high load environment": - So what exactly does it do in a "high load environment"? How do you define "high load" in the first place? CPU load? I/O load from other processes? What shortcomings of the competition under a "high load environment" are you trying overcome? Backup: - How do I do hot backup? Benchmark: - LevelDB is not a fair comparison as it offers additional non-trivial functionality (snapshots) that cannot be built up on top of Sophia. LevelDB APIs are also safe for concurrent use, which adds overhead. Kyoto Cabinet would have been more suitable as a peer to benchmark with. - 3 million records with 16-byte keys and 100-byte values is not really an interesting benchmark dataset. - Iteration over a static database is not interesting, either. Is there any alternative other than locking an entire mutating database for the duration of iteration?
- oscargrouch 13y agoReally guys, can you give more constructive or at least more (not based in bullshit assumptions) comments? if not, just shut up.. This is a non-trivial effort, and all people do is to complain about the font face or if the punctuation was right? First, in the benchmarks it just crush leveldb, this is already by itself a great achievement. can you confront the benchmarks? you do it one yourself with a different configuration? no? Second, if you are not a database expert and can create proper critics (constructive or not), just keep it to yourself.. i wonder how so many people get up with all of this conclusions so fast, without a proper look at the source code and to have a reasonable amount of time to know what are they talking about. its very hard to create things like this, but very easy to critisize without any background.. dont forget about it if you have something to say about a small thing, that do not have a direct relation to the product or thing itself, if theres already one comment about it, that enough! do not spam, answering it, or creating new comments about it, this is just so rude and unrespectful.. really, things are getting creepy on HN.. and its not only in this thread
- VikingCoder 13y agoYou: Making the documentation readable and easy to parse adds no value to projects! Everyone who disagrees should shut up. If I'm being kind to you, HN commenters (myself included) should do a better job of commenting politely, and spend more effort making sure their criticism comes off as constructive rather than just whining and aggressive... ...but I think you make it sound like criticism of anything outside of the source code itself is creepy, rude, and disrespectful.