7 ms·
LevelDB: A Fast Persistent Key-Value Store
The inventors being Jeff Dean and Sanjay Ghemawat.
More here : https://plus.google.com/118227548810368513262/posts/1UEtSkKp1vv
- trungonnews 15y agohow is this different from membase?
- eis 15y agoUhm shouldn't that be ovious by reading the high level descriptions of each? They are for completely different use cases. Membase is a distributed Key/Value server and LevelDB is a Key/Value library.
- thomas11 15y agoI see the difference between a server and a library, but both can often be used for the same use case. Just recently I evaluated a few data stores for a project and I didn't care all that much about the distinction. For the servers you're gonna use an API for your programming language anyway, so the programming model isn't that different.
- stonemetal 15y agoAbout like the difference between mysql and sqlite.
- timr 15y agohttp://www.hnsearch.com/search#request/all&q=leveldb http://www.hnsearch.com/search#request/all&q=leveldb
- swah 15y agoYour point?
- timr 15y agoThat leveldb has been discussed several times on HN in the last two months. I just didn't break out the links from the search UI. Downvoters: links to previous context are generally considered a good thing here.
- swah 15y agoOh, then its easier to understand that in this query: http://www.hnsearch.com/search#request/all&q=leveldb&sortby=create_ts+asc http://www.hnsearch.com/search#request/all&q=leveldb&...
- timr 15y agoYeah, the search results when I linked them had those results. The index was subsequently updated, so the top hits are now this thread, and this article. Oops.
- deleted 15y ago[deleted]
- dchest 15y agoSee also: Previous discussion - http://news.ycombinator.com/item?id=2526032 http://news.ycombinator.com/item?id=2526032 Benchmarks vs Kyoto TreeDB and SQLite3 - http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html (discussion - http://news.ycombinator.com/item?id=2813061 http://news.ycombinator.com/item?id=2813061) Benchmarks vs InnoDB - http://blog.basho.com/2011/07/01/Leveling-the-Field/ http://blog.basho.com/2011/07/01/Leveling-the-Field/
- rektide 15y agoOpen sourced as of March 18th, 2011: http://code.google.com/p/leveldb/source/browse/trunk/LICENSE?spec=svn2&r=2 http://code.google.com/p/leveldb/source/browse/trunk/LICENSE... That initial checkin: http://code.google.com/p/leveldb/source/detail?r=2 http://code.google.com/p/leveldb/source/detail?r=2
- gaborcselle 15y agoYes, we put up the Google Code site incognito mode back then, but have since added a number of bugfixes and optimizations, so we're actually comfortable announcing the project now.
- skanuj 15y agoNote to myself : Search before you post. Apologies, I checked new and front page only!
- dchest 15y agoYou were right to post it -- it's a new blog post announcing the non-beta release and benchmarks.
- gaborcselle 15y agoHi there! I'm a YC alum (reMail W09) and helped Jeff and Sanjay with LevelDB. Let me know if you have any questions about LevelDB and I'll see if I can help.
- clumsysmurf 15y agoI would really like to use this from my Android Java application. Is this possible, and what would be the best way to accomplish this?
- gojomo 15y agoAny comparisons of performance or functionality against BerkeleyDB?
- sallen 15y agoDoes this system have transactions and ACID guarantees?
- aashay 15y agoHow does this compare to other persistent key-value stores such as Membase?
- dlsspy 15y agoMembase is a clustered data storage service your application uses. LevelDB is a persistence library. That makes LevelDB the kind of thing you plug into membase to get the unique properties it has to offer (or at least for fun).
- pjscott 15y agoIn fact, the Riak guys are planning on doing exactly that: offering LevelDB as one of the storage back-ends, perhaps even the default. http://blog.basho.com/2011/07/01/Leveling-the-Field/ http://blog.basho.com/2011/07/01/Leveling-the-Field/
- jcapote 15y agoit would be cool to make a leveldb backed fork of redis
- stanleydrew 15y agoPardon my ignorance but what's backing redis currently?
- stephth 15y agoRedis. :) But more importantly, it runs as a server. I think what @jcapote meant is being able to use Redis operations without a server, like Leveldb or sqlite. I would love to see that. There's already some effort towards that direction [1], but using a google backed project instead of building a full library from the ground up could be a saner approach. [1] https://github.com/seppo0010/redislite https://github.com/seppo0010/redislite
- deleted 15y ago[deleted]
- swah 15y agoSo that's what Jeff does!
- swah 15y agoInteresting how, like in the open-sourced protobuf, there are no commits by Jeff or Sanjay...
- shadowmatter 15y agoJeff and Sanjay wrote the original protocol buffer implementation. The project was taken over by Kenton Varda, who rewrote the C++ and Java parts; this is what was open sourced. See http://temporal.fateofio.org/files/resume http://temporal.fateofio.org/files/resume
- swah 15y agoWhich is what I point as being interesting.
- gaborcselle 15y agodgrogan and myself have been batching changes to LevelDB from our internal code repository to put them on the Google Code page. Playing Google Code site admin didn't seem to me like a good use of Jeff and Sanjay's time.
- swah 15y agoYep, its great you guys could separate it from internal dependencies! Congrats.
- newhouseb 15y agoHow is this different than BDB?
- stephth 15y agoyou don't need to pay anyone to use it in your commercial software.
- newhouseb 15y agoI suppose if you are shipping proprietary binaries, then yes. But otherwise it's effectively GPL'ed.
- stephth 15y agoExactly. BDB is GPL with an (non-publicly priced) commercial license to embed in apps, and LevelDB is BSD. With LevelDb, one of the most practical use case of a serverless db - embedding it in a client-side application - isn't crippled by it's own license.
- davidhollander 15y agoBDB is a key\value store for unordered data more similar to Tokyo Cabinet hash databases. Tokyo Cabinet hash databases are a much faster option than BDB if you only need unordered data. LevelDB is for if you need ordered data, and a more appropriate comparison would be against a B+\tree database.
- stephth 15y agoLevelDB is for if you need ordered data LevelDB is slower with random reads, but that doesn't mean you shouldn't use it for unordered data - it's still quite fast.
- davidhollander 15y ago>LevelDB is slower with random reads, but that doesn't mean you shouldn't use it for unordered data - it's still quite fast. In a positive analysis (should rather than shouldn't), assuming no default choice, it seems rational to use Tokyo Cabinet or CDB _hashmaps_ for unordered data, and LevelDB for ordered data, from a datastructure and performance standpoint. To assert more would probably need a specific use case for context.
- gojomo 15y agoAn interesting development a while back that I'm surprised hasn't received more attention was Oracle's release of a SQLite-based interface to BDB: http://www.oracle.com/technetwork/database/berkeleydb/overview/sql-160887.html http://www.oracle.com/technetwork/database/berkeleydb/overvi... It's essentially drop-in compatible with SQLite, but with added concurrency and speed for most operations. (The concurrency addresses a major issue usually keeping SQLite as a prototyping/single-user-only option in web development.) With LevelDB as a BSD-licensed alternative to BDB, I wonder: (1) How would the LevelDB-vs-SQLite benchmarks change against SQLite+BDB backend? (2) Could a SQLite fork with a LevelDB backend get a performance boost?
- gaborcselle 15y agoThanks for the link! You could theoretically just compile this file against SQLite-based BDB: http://code.google.com/p/leveldb/source/browse/trunk/doc/bench/db_bench_sqlite3.cc http://code.google.com/p/leveldb/source/browse/trunk/doc/ben... And get the numbers yourself. (If you do, please post them here.)
- est 15y ago> SQLite-based interface to BDB One thing I didn't get about SQL API for BDB, how does something like select * from users where name!='tom' work ?
- gojomo 15y agoYou really don't know or care that you're using BDB; it works (to the user) just like SQLite. (Behind the scenes, it's using BDB for the tables/indexes, and so would do various full- or partial- table scans much like SQLite's native on-disk format.)
- taylorbuley 15y agoUpcoming versions of the Chrome browser include an implementation of the IndexedDB HTML5 API that is built on top of LevelDB Really excited about seeing IndexedDB run atop of this
- gleb 15y agoThe synchronous writes benchmark is interesting. This is normally bound by # seeks your disk can do per second, which is mostly a function of rotational speed. With 7200RPM drive you get 7200/60 = 120 of these a second. So the 100 and 110 numbers for competitors make sense. 2,400 for LevelDB does not. Is LevelDB batching writes or is there something more interesting going on?
- groby_b 15y agoIf you write full disk blocks, wouldn't the disk cache hide the seek latency?
- gleb 15y agoHaving write disk cache on would certainly explain it. But that leaves the question of discrepancy with numbers with competitors. You turn off write-through caching on disks when you run a database unless you are willing to accept corruption (which is worse than data loss) on power outage. And that's why you can't get acceptable write performance out of database without a battery-backed RAID controller (or something other kind of RAM-based write cache with a battery backup). Here's a simple way to test # fsyncs/s (a.k.a. commit rate) on your system: sysbench --test=fileio --file-fsync-freq=1 --file-num=1 \ --file-total-size=16384 --file-test-mode=rndwr run --max-time=10 \ | grep "Requests/sec"
- groby_b 15y agoIf cache is on, any performance discrepancy can be explained away by "usage patterns" :) Also, do you really mean turn off write-through, or did you mean write-behind? (I can't see how write-through would cause corruption, but maybe I'm missing something...) Also, I wouldn't be surprised if there's a discrepancy in the flushing code across systems. God knows flushing a file to disk in cross-platform code is an arcane science :) And finally, as somebody else pointed out, LevelDB seems to order write access sequentially as much as possible.
- stephth 15y agoIs LevelDB batching writes Yes, updates can be done in one atomic batch. Please correct me if I'm wrong, but I don't think Tokyo Cabinet allows it without Tokyo Tyrant.
- jchrisa 15y agoBindings for Node.js if anyone is interested: https://github.com/creationix/node-leveldb https://github.com/creationix/node-leveldb
- stephth 15y agoFrom the announcement: it has already been ported to a variety of Unix based systems, Mac OS X, Windows, and Android. It's worth noting that the makefile includes options to build for iOS. I've successfully done it and my next iOS app will include LevelDB. Also worth noting, thanks to the iOS devices SSDs, it's much faster than with the traditional HDD machines.
- mumrah 15y agoAnyone know how LevelDB compares to Voldemort? From a cursory glance, they are identical in their simple API (get, put, delete)
- gaborcselle 15y agoMy understanding is that Voldemort is a distributed key-value storage system, while LevelDB is a local on-disk key-value storage system.
- strlen 15y agoVoldemort developer here-- Voldemort to LevelDB is what MySQL is to InnoDB: Voldemort is a distributed system that allows multiple engines to be plugged in. Mostly commonly, companies use either BerkeleyDB or MySQL as a storage engine. LinkedIn, Mendeley, EBay and others also use the read only storage engine, where the data is pre-built in Hadoop and loaded into Voldemort. I am really excited about LevelDB: while there are higher priority projects on my plate right now, we'd very much like to see a LevelDB storage engine. If anyone is interested in contributing one, they're welcome. The steps are: 1) Creating JNI bindings to LevelDB (or creating a .so version of LevelDB and creating JNA bindings) 2) Implementing the StorageEngine interface with the bindings, including passing in any configuration. Here is an example of a third party InnoDB/Haildb storage engine for Voldemort: https://github.com/sunnygleason/v-storage-haildb https://github.com/sunnygleason/v-storage-haildb
- swah 15y agoI love the insight about how fast compression (Snappy) is like having faster hard drives.
- jadeddev 15y agoAs many of the comments are, "how does it compare to xyz?", the real question is, why the fuck do we need another key-value store?
- stonemetal 15y agoSorry if this is a bit off topic but it seems to me like most of Google's opensource projects are more source available than open source. Do they actually take contributions from the community or are they all like android, source made available when its "ready for public consumption"? LevelDB sounds like something I would like to contribute to but if the reception is going to be chilly I won't bother, maybe pick up mongo or redis instead.
- overred 15y agoLSM-Tree is good!
- newman314 15y agoHow well does LevelDB work for a mobile device? This might be a nice use case.