5 ms·

Ask HN: What are the best key-value self-hosted storage engines?

It could be a full db or a file-based binary. Interested in being: - simple - thread safe - fast - supporting lots of records - querying features is a plus

PaulHoule 5y ago
In Java I like https://mapdb.org/ https://mapdb.org/ It is more feature rich than you want but in Python I'd probably just use sqlite3 since it is in the standard library.
throwaway888abc 5y ago
https://redis.io/ https://redis.io/
inphovore 5y ago
http://rocksdb.org/ http://rocksdb.org/
sethammons 5y ago
Tossing another vote in for Redis. Generally, you have to give me a reason why Redis is not the answer for a KV store situation. It can run in a cluster mode or individually, and it can choose to run only in memory and not sync to disk if you don't need persistence and you want it fast. Some folks claim 200k inserts a second, but we topped out at around 20k inserts a second and had to start digging into more advanced configurations and whatnot. Eventually we sharded and sent some requests to one pool of redis and other requests to a different pool based on consistent hashing that we set up. But even then, I don't think most shops need to worry about that scale.
- aynyc 5y ago
  Is Redis fully ACID? We always used it as a caching engine.
- yuppie_scum 5y ago
  As I recall it’s a bitch making redis highly available in production. But maybe easier now if you find a good K8s operator for it or just use elasticache.
eatonphil 5y ago
Since Redis and memcached are historically the defacto key-value databases I'm curious what about them makes them not as obvious a choice for you. I'm not saying everyone should only use those two I'm just guessing there's more to the criteria you have in mind than you wrote here.
bitobserver 5y ago
Check out Skytable - https://skytable.io/ https://skytable.io/ Pros: Extremely fast. https://github.com/ohsayan/sky-benches https://github.com/ohsayan/sky-benches Cons: Still early in development, not a ton of features.
- avinassh 5y ago
  Redis shines at latency and I don’t see any benchmarks. Ofcourse Skytable throughput will be higher cos it’s multithreaded. I don’t think this is a fair benchmark
diehunde 5y ago
LevelDB: https://github.com/google/leveldb https://github.com/google/leveldb
toast0 5y ago
Mnesia from OTP (Erlang)
deleted 5y ago
[deleted]
endisneigh 5y ago
FoundationDB

It could be a full db or a file-based binary. Interested in being: - simple - thread safe - fast - supporting lots of records - querying features is a plus

PaulHoule 5y ago

In Java I like https://mapdb.org/ https://mapdb.org/ It is more feature rich than you want but in Python I'd probably just use sqlite3 since it is in the standard library.

throwaway888abc 5y ago

https://redis.io/ https://redis.io/

inphovore 5y ago

http://rocksdb.org/ http://rocksdb.org/

sethammons 5y ago

Tossing another vote in for Redis. Generally, you have to give me a reason why Redis is not the answer for a KV store situation. It can run in a cluster mode or individually, and it can choose to run only in memory and not sync to disk if you don't need persistence and you want it fast. Some folks claim 200k inserts a second, but we topped out at around 20k inserts a second and had to start digging into more advanced configurations and whatnot. Eventually we sharded and sent some requests to one pool of redis and other requests to a different pool based on consistent hashing that we set up. But even then, I don't think most shops need to worry about that scale.

aynyc 5y ago
Is Redis fully ACID? We always used it as a caching engine.
yuppie_scum 5y ago
As I recall it’s a bitch making redis highly available in production. But maybe easier now if you find a good K8s operator for it or just use elasticache.

eatonphil 5y ago

Since Redis and memcached are historically the defacto key-value databases I'm curious what about them makes them not as obvious a choice for you. I'm not saying everyone should only use those two I'm just guessing there's more to the criteria you have in mind than you wrote here.

bitobserver 5y ago

Check out Skytable - https://skytable.io/ https://skytable.io/ Pros: Extremely fast. https://github.com/ohsayan/sky-benches https://github.com/ohsayan/sky-benches Cons: Still early in development, not a ton of features.

avinassh 5y ago
Redis shines at latency and I don’t see any benchmarks. Ofcourse Skytable throughput will be higher cos it’s multithreaded. I don’t think this is a fair benchmark

diehunde 5y ago

LevelDB: https://github.com/google/leveldb https://github.com/google/leveldb

toast0 5y ago

Mnesia from OTP (Erlang)

deleted 5y ago

[deleted]

endisneigh 5y ago

FoundationDB