4 ms·

Cassandra is JVM GC issues Redis is in-memory only Cloud Storage - Not sure, how we can use it outside of cloud vendors RocksDB - Facebook just outsourced t

Cassandra is JVM GC issues Redis is in-memory only Cloud Storage - Not sure, how we can use it outside of cloud vendors RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? I am sure, they use one internally, why is it not being open-sourced? There is also badger but most of these only offer low-level operation. Sorry, most of my developers are unable to consume them just like Redis.

kodablah 9y ago
Agree w/ all these points (forgot "S3 has provider lock-in" though others replicate the API). I do use Cassandra for most use cases and I don't hit GC things, but I understand the concerns that come with the JVM (no, haven't subbed Scylla in yet). One that I haven't tried but want to hear the downsides of is https://github.com/pingcap/tikv https://github.com/pingcap/tikv (not the DB built on top, but just that one for KV). A nice published list of all cons of all database systems would be ideal.
- SanFranManDan 9y ago
  From what I could tell, its tikv is virtually useless without the rest of the tidb (the MySQL layer that sits on top written in go) to be useful. TiKV has no replication / sharding built in, that is actually handled by a Placement Driver (PD) From the docs: > TiKV is a component in the TiDB project, you must build and run it with TiDB and PD together.
majidazimi 9y ago
> where is the service which adds replication, clustering and network interface on top of it? Actually I would prefer such kind of database (just the engine). 1. Expose it to network via what ever framework you like. Thrift, Rest, grpc, ... You don't have to include different kind of libraries for each network service. I would love to connect to every network service (Redis, Elasticsearch, Cassandra, MySQL, ...) via a single framework (say grpc). 2. In most large scale scenarios, there is already some kind of log service (DistributedLog, NATS, Kafka, ...). Why not take benefit of that for replication? Isn't it great to separate the engine layer from replication layer? Currently we are doing double replication actually. Replicate data from master DB to slave DB. Then replicate the same data, from any DB to cache, search, ... components. The data is already there on log. Let everyone (slave DB as well as cache/search module) consume it. This is basically state machine replication idiom. PNUTS[0], Twitter K/V database, LinkedIn Espresso [1] (as well as Ambry[2] which is their internal object store), ... use this approach for replication. 3. I would agree with that, they only support basic low level operations. [0] http://www.vldb.org/pvldb/1/1454167.pdf http://www.vldb.org/pvldb/1/1454167.pdf [1] http://www.csce.uark.edu/~xintaowu/BDAM/p1135-qiao.pdf http://www.csce.uark.edu/~xintaowu/BDAM/p1135-qiao.pdf [2] http://dprg.cs.uiuc.edu/docs/SIGMOD2016-a/ambry.pdf http://dprg.cs.uiuc.edu/docs/SIGMOD2016-a/ambry.pdf
OhDagny 9y ago
Cassandra GC: check out scylla or the rocksandra projects.
bpicolo 9y ago
You can disk-back redis at various consistency levels if you so choose https://redis.io/topics/persistence https://redis.io/topics/persistence
nilkn 9y ago
Redis supports at least two forms of persistence: (1) periodic checkpoints; (2) writing an append-only log which can be periodically compacted to save space.
electrum 9y ago
> RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? https://myrocks.io/ https://myrocks.io/
petre 9y ago
They also released MyRocks, a RocksDB engine for MySQL which will is being intergrated in MariaDB 10.3 and Percona Server 5.7. I don't think it supports replication yet, but it's a start.
zilchers 9y ago
Cassandra has a host of other trade offs besides java (tombstones and data garbage collection come to mind).
- mspoof 9y ago
  Rocks Storage engine in Cassandra which claims to reduce the GC stalls https://engineering.instagram.com/open-sourcing-a-10x-reduction-in-apache-cassandra-tail-latency-d64f86b43589 https://engineering.instagram.com/open-sourcing-a-10x-reduct...

kodablah 9y ago

Agree w/ all these points (forgot "S3 has provider lock-in" though others replicate the API). I do use Cassandra for most use cases and I don't hit GC things, but I understand the concerns that come with the JVM (no, haven't subbed Scylla in yet). One that I haven't tried but want to hear the downsides of is https://github.com/pingcap/tikv https://github.com/pingcap/tikv (not the DB built on top, but just that one for KV). A nice published list of all cons of all database systems would be ideal.

SanFranManDan 9y ago
From what I could tell, its tikv is virtually useless without the rest of the tidb (the MySQL layer that sits on top written in go) to be useful. TiKV has no replication / sharding built in, that is actually handled by a Placement Driver (PD) From the docs: > TiKV is a component in the TiDB project, you must build and run it with TiDB and PD together.

majidazimi 9y ago

> where is the service which adds replication, clustering and network interface on top of it? Actually I would prefer such kind of database (just the engine). 1. Expose it to network via what ever framework you like. Thrift, Rest, grpc, ... You don't have to include different kind of libraries for each network service. I would love to connect to every network service (Redis, Elasticsearch, Cassandra, MySQL, ...) via a single framework (say grpc). 2. In most large scale scenarios, there is already some kind of log service (DistributedLog, NATS, Kafka, ...). Why not take benefit of that for replication? Isn't it great to separate the engine layer from replication layer? Currently we are doing double replication actually. Replicate data from master DB to slave DB. Then replicate the same data, from any DB to cache, search, ... components. The data is already there on log. Let everyone (slave DB as well as cache/search module) consume it. This is basically state machine replication idiom. PNUTS[0], Twitter K/V database, LinkedIn Espresso [1] (as well as Ambry[2] which is their internal object store), ... use this approach for replication. 3. I would agree with that, they only support basic low level operations. [0] http://www.vldb.org/pvldb/1/1454167.pdf http://www.vldb.org/pvldb/1/1454167.pdf [1] http://www.csce.uark.edu/~xintaowu/BDAM/p1135-qiao.pdf http://www.csce.uark.edu/~xintaowu/BDAM/p1135-qiao.pdf [2] http://dprg.cs.uiuc.edu/docs/SIGMOD2016-a/ambry.pdf http://dprg.cs.uiuc.edu/docs/SIGMOD2016-a/ambry.pdf

OhDagny 9y ago

Cassandra GC: check out scylla or the rocksandra projects.

bpicolo 9y ago

You can disk-back redis at various consistency levels if you so choose https://redis.io/topics/persistence https://redis.io/topics/persistence

nilkn 9y ago

Redis supports at least two forms of persistence: (1) periodic checkpoints; (2) writing an append-only log which can be periodically compacted to save space.

electrum 9y ago

> RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? https://myrocks.io/ https://myrocks.io/

petre 9y ago

They also released MyRocks, a RocksDB engine for MySQL which will is being intergrated in MariaDB 10.3 and Percona Server 5.7. I don't think it supports replication yet, but it's a start.

zilchers 9y ago

Cassandra has a host of other trade offs besides java (tombstones and data garbage collection come to mind).

mspoof 9y ago
Rocks Storage engine in Cassandra which claims to reduce the GC stalls https://engineering.instagram.com/open-sourcing-a-10x-reduction-in-apache-cassandra-tail-latency-d64f86b43589 https://engineering.instagram.com/open-sourcing-a-10x-reduct...