6 ms·

Show HN: RedixDB – a fast persistent key-value store with the Redis protocol

75 points by alash3al 8y ago

volument 8y ago
Do you plan to support sentinel and replication?
- alash3al 8y ago
  I started redix with redis protocol to make use of redis ecosystem out there. There is also a plan to make HA a native feature in redix with no external requirements, so I can keep redix a standalone DB server.
cperciva 8y ago
A somewhat more accurate title, in case it saves someone else the time I just spent: "RedixDB - a Redis frontend to the Badger key-value store".
- alash3al 8y ago
  So, mysql is a frontend to InnoDB? Cockroach is a frontend to RocksDB? Note: Redix has a pluggable storage engine, the current engine is "badgerdb", the next is "bolt" and it is already integrated and I'll release it soon.
  - cperciva 8y ago
    So, mysql is a frontend to InnoDB? I would put it the other way around: InnoDB is a backend for MySQL. But an SQL database is far more complicated than a key-value store, and has a lot of "middle end" between the "talk to the client" front end and the "store stuff on disk" back end. I'm not saying that RedixDB isn't interesting -- having a network front-end to Badger seems like it would extremely useful for most people who want to use Badger. It's just that I spent a while digging through the code trying to figure out where the back end was and what data structures you were using, only to discover that you didn't have a backend (or rather, you're using a pre-existing one).
    - throwaway12iii 8y ago
      Your second comment is less friendly than your first. Maybe just stop.
    - kieckerjan 8y ago
      Second that. It would help if you included on the website some info about the technical implementation. That would help instill confidence in the solution.
  - networkimprov 8y ago
    None of these backends currently fsync() correctly on MacOS, due to a Go bug fixed in 1.12.
    - fulafel 8y ago
      To split hairs, the bug is in MacOS - its fsync() does not flush data to stable storage. Go 1.12 includes a workaround for this MacOS bug.
      - networkimprov 8y ago
        To split further, it's a MacOS design flaw, with a published workaround. It is a bug to fail to use the workaround :-)
        fulafel 8y ago
        I don't know if Apoke really acknowledges it as a bug or flaw. Their man page for fsync reads like an intentional misunderstanding of fsync semantics.
  - rlaanemets 8y ago
    There is also Facebook's MyRocks which is MySQL frontend with RocksDB backend. It has lower write amplification than InnoDB, which is the main reason why it was created.
- tidwall 8y ago
  RedixDB appears to provide a lot of new functionality that is not available to a plain Badger database. I disagree that Badger plays such a big part in this project that it needs to be included in the title. Badger is only the storage component and it can be easily swapped out for an alternative.
etaioinshrdlu 8y ago
Do you suppose it will ever be possible to support transactions with MULTI? Is the DB strongly consistent? Are there performance gains compared to Redis due to multi-core?
- alash3al 8y ago
  1- Multi: yes, I'm planning to support it in the next releases. 2- Consistency: Yep, also there will be test cases soon. 3- Performance: at first, nothing will be faster than RAM, so don't expect that an on-disk store is faster than an in-memory store, but that doesn't mean that on-disk stores are very slow, I selected two storage engines called bolt and badger, badger is the default because it follows RocksDB design and rocksdb. for redix server compared to redis server, redix doesn't use an eventloop, but each connection runs in its own light-weight thread, so there no possibilities that you run a command and block the server itself, another note is: the underlying implementation of internal datatypes makes redix faster in some cases especially when you try to load large datasets, you will notice that redix has very low response time than redis, that means, in a highly loaded environment you find redix performance better than redis especially with the coming version because redix will lower the memory usage more and more. Every day I improve it more and more, and thanks for the issues creator on the repo, they're helping me too in organizing my thoughts.
  - e12e 8y ago
    Any suggested/safe backup/restore strategy?
    - alash3al 8y ago
      Currently I didn't create commands for that, but you can easly do the following: - Backup: Copy the storage directory as it. - Restore: stop the server, move the previously backup data directly to redix storage. It is very simple, but I may add commands that will help you with that operation.
  - koolba 8y ago
    > for redix server compared to redis server, redix doesn't use an eventloop, but each connection runs in its own light-weight thread, so there no possibilities that you run a command and block the server itself, How does this jive with the “happens-before” transactional guarantees of Redis? Since it’s single threaded it’s easy to reason about the order of operations. Once an operation begins subsequent ones can assume the full impact of the changes. In Redix, if a “big write” happens to a key in one connection/thread, what does a different reader see for the value of that key if they issue a read after the write begins but before it completes? The old value or a wait followed by the new value?
siddontang 8y ago
Cool, I did this several years ago. You can check LedisDB or RebornDB. But now I switch to TiKV, hope you can support it later :-)
- alash3al 8y ago
  Thank you ^_^, Greg Weber from PingCap mailed me and I told him that I'm planning to support TiKV :)
manigandham 8y ago
Nice work on this. I wish more databases used the Redis protocol as it would simplify the numerous different interfaces out there. There are so many key/value systems out there yet many that aren't very usable outside of specific embedded use-cases.
sertorius 8y ago
Thanks for this! Is there a suggested way to get a list of keys based on wildcard? I note that the infamous KEYS command is not listed as supported (and fair enough). Is there a suitable non-Redis backend command that can get this info?
- alash3al 8y ago
  Noted, and will do it or a similar one in the next release :), v1.6 is now released, and 1.7 will include a support for your case :)
coleifer 8y ago
Badger is an lsm-tree storage engine which makes some unusual tradeoffs. Good background info here: https://blog.dgraph.io/post/badger/ https://blog.dgraph.io/post/badger/
gregwebs 8y ago
Relatedly, there are some active efforts to make a Redis on top of TiKV, which is a distributed persistent Key-Value store with transactions. Titan: https://medium.com/@shafreeck/titan-a-distributed-redis-protocol-compatible-nosql-database-4379a1307661 https://medium.com/@shafreeck/titan-a-distributed-redis-prot... Tidis: https://github.com/yongman/tidis https://github.com/yongman/tidis

75 points by alash3al 8y ago

volument 8y ago

Do you plan to support sentinel and replication?

alash3al 8y ago
I started redix with redis protocol to make use of redis ecosystem out there. There is also a plan to make HA a native feature in redix with no external requirements, so I can keep redix a standalone DB server.

cperciva 8y ago

A somewhat more accurate title, in case it saves someone else the time I just spent: "RedixDB - a Redis frontend to the Badger key-value store".

alash3al 8y ago
So, mysql is a frontend to InnoDB? Cockroach is a frontend to RocksDB? Note: Redix has a pluggable storage engine, the current engine is "badgerdb", the next is "bolt" and it is already integrated and I'll release it soon.
- cperciva 8y ago
  So, mysql is a frontend to InnoDB? I would put it the other way around: InnoDB is a backend for MySQL. But an SQL database is far more complicated than a key-value store, and has a lot of "middle end" between the "talk to the client" front end and the "store stuff on disk" back end. I'm not saying that RedixDB isn't interesting -- having a network front-end to Badger seems like it would extremely useful for most people who want to use Badger. It's just that I spent a while digging through the code trying to figure out where the back end was and what data structures you were using, only to discover that you didn't have a backend (or rather, you're using a pre-existing one).
  - throwaway12iii 8y ago
    Your second comment is less friendly than your first. Maybe just stop.
  - kieckerjan 8y ago
    Second that. It would help if you included on the website some info about the technical implementation. That would help instill confidence in the solution.
- networkimprov 8y ago
  None of these backends currently fsync() correctly on MacOS, due to a Go bug fixed in 1.12.
  - fulafel 8y ago
    To split hairs, the bug is in MacOS - its fsync() does not flush data to stable storage. Go 1.12 includes a workaround for this MacOS bug.
    - networkimprov 8y ago
      To split further, it's a MacOS design flaw, with a published workaround. It is a bug to fail to use the workaround :-)
      - fulafel 8y ago
        I don't know if Apoke really acknowledges it as a bug or flaw. Their man page for fsync reads like an intentional misunderstanding of fsync semantics.
- rlaanemets 8y ago
  There is also Facebook's MyRocks which is MySQL frontend with RocksDB backend. It has lower write amplification than InnoDB, which is the main reason why it was created.
tidwall 8y ago
RedixDB appears to provide a lot of new functionality that is not available to a plain Badger database. I disagree that Badger plays such a big part in this project that it needs to be included in the title. Badger is only the storage component and it can be easily swapped out for an alternative.

etaioinshrdlu 8y ago

Do you suppose it will ever be possible to support transactions with MULTI? Is the DB strongly consistent? Are there performance gains compared to Redis due to multi-core?

alash3al 8y ago
1- Multi: yes, I'm planning to support it in the next releases. 2- Consistency: Yep, also there will be test cases soon. 3- Performance: at first, nothing will be faster than RAM, so don't expect that an on-disk store is faster than an in-memory store, but that doesn't mean that on-disk stores are very slow, I selected two storage engines called bolt and badger, badger is the default because it follows RocksDB design and rocksdb. for redix server compared to redis server, redix doesn't use an eventloop, but each connection runs in its own light-weight thread, so there no possibilities that you run a command and block the server itself, another note is: the underlying implementation of internal datatypes makes redix faster in some cases especially when you try to load large datasets, you will notice that redix has very low response time than redis, that means, in a highly loaded environment you find redix performance better than redis especially with the coming version because redix will lower the memory usage more and more. Every day I improve it more and more, and thanks for the issues creator on the repo, they're helping me too in organizing my thoughts.
- e12e 8y ago
  Any suggested/safe backup/restore strategy?
  - alash3al 8y ago
    Currently I didn't create commands for that, but you can easly do the following: - Backup: Copy the storage directory as it. - Restore: stop the server, move the previously backup data directly to redix storage. It is very simple, but I may add commands that will help you with that operation.
- koolba 8y ago
  > for redix server compared to redis server, redix doesn't use an eventloop, but each connection runs in its own light-weight thread, so there no possibilities that you run a command and block the server itself, How does this jive with the “happens-before” transactional guarantees of Redis? Since it’s single threaded it’s easy to reason about the order of operations. Once an operation begins subsequent ones can assume the full impact of the changes. In Redix, if a “big write” happens to a key in one connection/thread, what does a different reader see for the value of that key if they issue a read after the write begins but before it completes? The old value or a wait followed by the new value?

siddontang 8y ago

Cool, I did this several years ago. You can check LedisDB or RebornDB. But now I switch to TiKV, hope you can support it later :-)

alash3al 8y ago
Thank you ^_^, Greg Weber from PingCap mailed me and I told him that I'm planning to support TiKV :)

manigandham 8y ago

Nice work on this. I wish more databases used the Redis protocol as it would simplify the numerous different interfaces out there. There are so many key/value systems out there yet many that aren't very usable outside of specific embedded use-cases.

sertorius 8y ago

Thanks for this! Is there a suggested way to get a list of keys based on wildcard? I note that the infamous KEYS command is not listed as supported (and fair enough). Is there a suitable non-Redis backend command that can get this info?

alash3al 8y ago
Noted, and will do it or a similar one in the next release :), v1.6 is now released, and 1.7 will include a support for your case :)

coleifer 8y ago

Badger is an lsm-tree storage engine which makes some unusual tradeoffs. Good background info here: https://blog.dgraph.io/post/badger/ https://blog.dgraph.io/post/badger/

gregwebs 8y ago

Relatedly, there are some active efforts to make a Redis on top of TiKV, which is a distributed persistent Key-Value store with transactions. Titan: https://medium.com/@shafreeck/titan-a-distributed-redis-protocol-compatible-nosql-database-4379a1307661 https://medium.com/@shafreeck/titan-a-distributed-redis-prot... Tidis: https://github.com/yongman/tidis https://github.com/yongman/tidis