4 ms·

Nice, I'm currently using SQLite for key value storage, because there aren't better light options afaik. Python shelve module is totally useless with multi giga

Nice, I'm currently using SQLite for key value storage, because there aren't better light options afaik. Python shelve module is totally useless with multi gigabyte tables and millions of keys.

monstrado 13y ago
Check into Google's LevelDB library.
gizzlon 13y ago
What's wrong with BerkleyDB, Tokyo Kabinet, Kyoto Kabinet, LevelDB .. ? It's a real question because I haven't actually used them. SQLite is fine, but if you don't need sql they probably fare better.
- Sami_Lehtinen 13y ago
  Afaik, all of those require additional setup. SQLite3 is already included in Python. I also tested year ago several database options and SQLite3 was also the fastest one.
  - josephg 13y ago
    Maybe it depends on your work load, but thats not true if you just want a regular K-V store. LevelDB and LMDB are usually way faster: http://symas.com/mdb/microbench/ http://symas.com/mdb/microbench/ Of course, neither of these systems have indexing. If you want to use them directly, you have to organise your data so that you can translate your queries into range queries on an ordered set. The FoundationDB guys talk about this here: http://foundationdb.com/documentation/beta1/data-modeling.html#indexes http://foundationdb.com/documentation/beta1/data-modeling.ht...
jonpaul 13y ago
Like another commenter, I was also going to recommend Google's LevelDB. It has bindings for just about every major language and it's my understanding it works really well as a KV store.
Shish2k 13y ago
Perhaps a little tangential, but that reminds me I wrote a SQLite interface with a shelf-like API, for when I wanted incredibly simple data storage which may grow more complex and relational later: from sqliteshelf import SQLiteShelf d = SQLiteShelf("filename.sdb", "tablename") # now use d like any other dictionary, but this one has # persistent storage https://github.com/shish/sqliteshelf https://github.com/shish/sqliteshelf

Nice, I'm currently using SQLite for key value storage, because there aren't better light options afaik. Python shelve module is totally useless with multi gigabyte tables and millions of keys.

monstrado 13y ago

Check into Google's LevelDB library.

gizzlon 13y ago

What's wrong with BerkleyDB, Tokyo Kabinet, Kyoto Kabinet, LevelDB .. ? It's a real question because I haven't actually used them. SQLite is fine, but if you don't need sql they probably fare better.

Sami_Lehtinen 13y ago
Afaik, all of those require additional setup. SQLite3 is already included in Python. I also tested year ago several database options and SQLite3 was also the fastest one.
- josephg 13y ago
  Maybe it depends on your work load, but thats not true if you just want a regular K-V store. LevelDB and LMDB are usually way faster: http://symas.com/mdb/microbench/ http://symas.com/mdb/microbench/ Of course, neither of these systems have indexing. If you want to use them directly, you have to organise your data so that you can translate your queries into range queries on an ordered set. The FoundationDB guys talk about this here: http://foundationdb.com/documentation/beta1/data-modeling.html#indexes http://foundationdb.com/documentation/beta1/data-modeling.ht...

jonpaul 13y ago

Like another commenter, I was also going to recommend Google's LevelDB. It has bindings for just about every major language and it's my understanding it works really well as a KV store.

Shish2k 13y ago

Perhaps a little tangential, but that reminds me I wrote a SQLite interface with a shelf-like API, for when I wanted incredibly simple data storage which may grow more complex and relational later: from sqliteshelf import SQLiteShelf d = SQLiteShelf("filename.sdb", "tablename") # now use d like any other dictionary, but this one has # persistent storage https://github.com/shish/sqliteshelf https://github.com/shish/sqliteshelf