5 ms·

Looks very very interesting. Nice work. Plans to support more DBs? do you intend for the OS community to build the translation layers?

by danr4 5y ago

losfair 5y ago
Thanks! > Plans to support more DBs? RocksDB and TiKV backends are planned: https://github.com/losfair/RefineDB/issues/3 https://github.com/losfair/RefineDB/issues/3 > do you intend for the OS community to build the translation layers? What do you mean by "translation layers" here?
- tluyben2 5y ago
  I really like TiKV: can you explain what the issue is with it? And do what are ways to resolve it if TiKV does not resolve it themselves? Did you do any profiling on your in memory store? I am looking, working on a toy project, for something like this that has 3 layers: local mem, distributed memory and distributed persistence with the same api and, more specifically, query engine. There are kv stores for these (TiKV and Foundation for the last category), Rondb maybe for the in distributed in memory but locally, it should be capable of very high volume (evmap doing a billion/minute but that would not work as not transactional). I bet it is too early to worry about perf for you but it seems you can leave most of that to the kv store which I like.
  - losfair 5y ago
    > can you explain what the issue is with it? And do what are ways to resolve it if TiKV does not resolve it themselves? TiKV is based on multi-raft + Percolator where there is no way to coordinate a consistent range operation across regions (key ranges) and therefore does not support range deletion. RefineDB uses range deletion to delete non-primitive types that contain multiple members, e.g. tables and sets. To support TiKV, RefineDB needs to support asynchronous range deletion - mark a range as deleted, and never use it again. This requires some work on the storage mapping layer, to map ABA accesses to a recreated-after-deleted range to a different keyspace. > Did you do any profiling on your in memory store? Not yet, actually the in-memory store is just for use in tests and the web playground where performance isn't important. For a better in-memory store, I think it's better to use the SQLite backend with an in-memory database.

Looks very very interesting. Nice work. Plans to support more DBs? do you intend for the OS community to build the translation layers?

by danr4 5y ago

Looks very very interesting. Nice work. Plans to support more DBs? do you intend for the OS community to build the translation layers?

losfair 5y ago

Thanks! > Plans to support more DBs? RocksDB and TiKV backends are planned: https://github.com/losfair/RefineDB/issues/3 https://github.com/losfair/RefineDB/issues/3 > do you intend for the OS community to build the translation layers? What do you mean by "translation layers" here?

tluyben2 5y ago
I really like TiKV: can you explain what the issue is with it? And do what are ways to resolve it if TiKV does not resolve it themselves? Did you do any profiling on your in memory store? I am looking, working on a toy project, for something like this that has 3 layers: local mem, distributed memory and distributed persistence with the same api and, more specifically, query engine. There are kv stores for these (TiKV and Foundation for the last category), Rondb maybe for the in distributed in memory but locally, it should be capable of very high volume (evmap doing a billion/minute but that would not work as not transactional). I bet it is too early to worry about perf for you but it seems you can leave most of that to the kv store which I like.
- losfair 5y ago
  > can you explain what the issue is with it? And do what are ways to resolve it if TiKV does not resolve it themselves? TiKV is based on multi-raft + Percolator where there is no way to coordinate a consistent range operation across regions (key ranges) and therefore does not support range deletion. RefineDB uses range deletion to delete non-primitive types that contain multiple members, e.g. tables and sets. To support TiKV, RefineDB needs to support asynchronous range deletion - mark a range as deleted, and never use it again. This requires some work on the storage mapping layer, to map ABA accesses to a recreated-after-deleted range to a different keyspace. > Did you do any profiling on your in memory store? Not yet, actually the in-memory store is just for use in tests and the web playground where performance isn't important. For a better in-memory store, I think it's better to use the SQLite backend with an in-memory database.