6 ms·

I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect i

I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect it to be. What the world of data needs more of is continued development into novel indexing strategies/implementations. ElasticSearch, Postgres's GIN index on JSONB, MapReduce, graph databases. I don't need another key value store...

mlevental 9y ago
there's a new one of those posted like every day - and every single one is "crazy fast" and "super scalable"
- michaelcampbell 9y ago
  Almost as frequent as the new hotness JS framework.
zellyn 9y ago
I don't mean this facetiously: did you read the article? This work presents novel ways of "solving" the problem that cause it to beat the other contenders fairly convincingly (in non-linearizable regimes).
xstartup 9y ago
Cassandra is JVM GC issues Redis is in-memory only Cloud Storage - Not sure, how we can use it outside of cloud vendors RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? I am sure, they use one internally, why is it not being open-sourced? There is also badger but most of these only offer low-level operation. Sorry, most of my developers are unable to consume them just like Redis.
- kodablah 9y ago
  Agree w/ all these points (forgot "S3 has provider lock-in" though others replicate the API). I do use Cassandra for most use cases and I don't hit GC things, but I understand the concerns that come with the JVM (no, haven't subbed Scylla in yet). One that I haven't tried but want to hear the downsides of is https://github.com/pingcap/tikv https://github.com/pingcap/tikv (not the DB built on top, but just that one for KV). A nice published list of all cons of all database systems would be ideal.
  - SanFranManDan 9y ago
    From what I could tell, its tikv is virtually useless without the rest of the tidb (the MySQL layer that sits on top written in go) to be useful. TiKV has no replication / sharding built in, that is actually handled by a Placement Driver (PD) From the docs: > TiKV is a component in the TiDB project, you must build and run it with TiDB and PD together.
- majidazimi 9y ago
  > where is the service which adds replication, clustering and network interface on top of it? Actually I would prefer such kind of database (just the engine). 1. Expose it to network via what ever framework you like. Thrift, Rest, grpc, ... You don't have to include different kind of libraries for each network service. I would love to connect to every network service (Redis, Elasticsearch, Cassandra, MySQL, ...) via a single framework (say grpc). 2. In most large scale scenarios, there is already some kind of log service (DistributedLog, NATS, Kafka, ...). Why not take benefit of that for replication? Isn't it great to separate the engine layer from replication layer? Currently we are doing double replication actually. Replicate data from master DB to slave DB. Then replicate the same data, from any DB to cache, search, ... components. The data is already there on log. Let everyone (slave DB as well as cache/search module) consume it. This is basically state machine replication idiom. PNUTS[0], Twitter K/V database, LinkedIn Espresso [1] (as well as Ambry[2] which is their internal object store), ... use this approach for replication. 3. I would agree with that, they only support basic low level operations. [0] http://www.vldb.org/pvldb/1/1454167.pdf http://www.vldb.org/pvldb/1/1454167.pdf [1] http://www.csce.uark.edu/~xintaowu/BDAM/p1135-qiao.pdf http://www.csce.uark.edu/~xintaowu/BDAM/p1135-qiao.pdf [2] http://dprg.cs.uiuc.edu/docs/SIGMOD2016-a/ambry.pdf http://dprg.cs.uiuc.edu/docs/SIGMOD2016-a/ambry.pdf
- OhDagny 9y ago
  Cassandra GC: check out scylla or the rocksandra projects.
- bpicolo 9y ago
  You can disk-back redis at various consistency levels if you so choose https://redis.io/topics/persistence https://redis.io/topics/persistence
- nilkn 9y ago
  Redis supports at least two forms of persistence: (1) periodic checkpoints; (2) writing an append-only log which can be periodically compacted to save space.
- electrum 9y ago
  > RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? https://myrocks.io/ https://myrocks.io/
- petre 9y ago
  They also released MyRocks, a RocksDB engine for MySQL which will is being intergrated in MariaDB 10.3 and Percona Server 5.7. I don't think it supports replication yet, but it's a start.
- zilchers 9y ago
  Cassandra has a host of other trade offs besides java (tombstones and data garbage collection come to mind).
  - mspoof 9y ago
    Rocks Storage engine in Cassandra which claims to reduce the GC stalls https://engineering.instagram.com/open-sourcing-a-10x-reduction-in-apache-cassandra-tail-latency-d64f86b43589 https://engineering.instagram.com/open-sourcing-a-10x-reduct...
SanFranManDan 9y ago
I agree. However, I stumbled into the world of KV stores (like RocksDB, LMDB, LevelDB, etc..) last year, and what is most surprising is that they all stop in the same place. I understand that they should do one thing and one thing well, but it is still disappointing when you have to implement things like replication, sharding, and indexing yourself. There really aren't even that many DBMS that are KV (like redis) out there to handle it either. They are normally much more complicated (like adding SQL layer on top of it).
- digikata 9y ago
  I think we're just getting to the point where it may become more common to separate the simpler problem of a single node, non-distributed 'store' with some kv interface below, and then build more complex distributed algorithms in a layer or two above. You can see some of the larger monolithic codebases that had to start by having their own code all the way up and down the stack, but now are starting to experiment with backend store interfaces so you can trade-off some of the strengths and weaknesses of various local store performance areas. Along the same lines, a few newer codebases for distributed stores seem to be building with those delineations in mind. Another comment brought up Tidb/Tikv for example. Tikv iirc uses RocksDB as its local store.
  - hyc_symas 9y ago
    "Just getting to the point" ? OpenLDAP has been architected this way for ~20 years. I think the same could be said for MySQL, as well as SQLServer (built on top of ESENT/JET). Large monolithic data stores are an obvious anti-pattern, reflects short-sighted design process.
    - digikata 9y ago
      Ha, you have a good point. I think my comment was directed more to "distributed cloud store" applications, much more so than databases. That begs the question - then what capabilities have those apps been focusing on that they didn't get from existing databases (or thought that they didn't get from existing databases)...
tracker1 9y ago
All the options you mention have varying features that give it advantages and disadvantages in various scenarios. What works well in on, won't in others, etc. That said, I was mostly interested on how this compares to etcd, since the use case seems pretty similar. How does Anna scale out (differing data centers) compared to it being a bad idea with etcd for example.
- doorbumper 9y ago
  I've just finished reading the paper. One difference is that Anna offers a weaker form of consistency than etcd. Etcd offers sequential consistency, and Anna offers casual consistency. I believe that this could eliminate many etcd use cases, since ordering for non-casual events many be different depending on the node you access. I'm not sure I fully understand etcd's durability guarantees. When an operation is completed, does that mean that the data is durable on a single machine, and then becomes durable on other machines at a later point? If so, it seems like Anna could offer the same durability guarantees. I think Anna's architecture could be more of a competitor to Cassandra, or DynamoDB as long as you only need casual consistency. The performance implications do seem pretty interesting.
  - jaricftw 9y ago
    etcd does not provide key partitioning/sharding, it achieves linearizable consistency using replica state machine via Raft. Anna provides weaker consistency guarantee (causal or read committed), but has partitioning/sharding.
naasking 9y ago
> I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect it to be This paper literally shows this statement is false. In what other circumstances could you provide one and two orders of magnitude performance improvement, and people would just shrug and say, "meh, I really don't think we need improved performance".

mlevental 9y ago

there's a new one of those posted like every day - and every single one is "crazy fast" and "super scalable"

michaelcampbell 9y ago
Almost as frequent as the new hotness JS framework.

zellyn 9y ago

I don't mean this facetiously: did you read the article? This work presents novel ways of "solving" the problem that cause it to beat the other contenders fairly convincingly (in non-linearizable regimes).

xstartup 9y ago

Cassandra is JVM GC issues Redis is in-memory only Cloud Storage - Not sure, how we can use it outside of cloud vendors RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? I am sure, they use one internally, why is it not being open-sourced? There is also badger but most of these only offer low-level operation. Sorry, most of my developers are unable to consume them just like Redis.

kodablah 9y ago
Agree w/ all these points (forgot "S3 has provider lock-in" though others replicate the API). I do use Cassandra for most use cases and I don't hit GC things, but I understand the concerns that come with the JVM (no, haven't subbed Scylla in yet). One that I haven't tried but want to hear the downsides of is https://github.com/pingcap/tikv https://github.com/pingcap/tikv (not the DB built on top, but just that one for KV). A nice published list of all cons of all database systems would be ideal.
- SanFranManDan 9y ago
  From what I could tell, its tikv is virtually useless without the rest of the tidb (the MySQL layer that sits on top written in go) to be useful. TiKV has no replication / sharding built in, that is actually handled by a Placement Driver (PD) From the docs: > TiKV is a component in the TiDB project, you must build and run it with TiDB and PD together.
majidazimi 9y ago
> where is the service which adds replication, clustering and network interface on top of it? Actually I would prefer such kind of database (just the engine). 1. Expose it to network via what ever framework you like. Thrift, Rest, grpc, ... You don't have to include different kind of libraries for each network service. I would love to connect to every network service (Redis, Elasticsearch, Cassandra, MySQL, ...) via a single framework (say grpc). 2. In most large scale scenarios, there is already some kind of log service (DistributedLog, NATS, Kafka, ...). Why not take benefit of that for replication? Isn't it great to separate the engine layer from replication layer? Currently we are doing double replication actually. Replicate data from master DB to slave DB. Then replicate the same data, from any DB to cache, search, ... components. The data is already there on log. Let everyone (slave DB as well as cache/search module) consume it. This is basically state machine replication idiom. PNUTS[0], Twitter K/V database, LinkedIn Espresso [1] (as well as Ambry[2] which is their internal object store), ... use this approach for replication. 3. I would agree with that, they only support basic low level operations. [0] http://www.vldb.org/pvldb/1/1454167.pdf http://www.vldb.org/pvldb/1/1454167.pdf [1] http://www.csce.uark.edu/~xintaowu/BDAM/p1135-qiao.pdf http://www.csce.uark.edu/~xintaowu/BDAM/p1135-qiao.pdf [2] http://dprg.cs.uiuc.edu/docs/SIGMOD2016-a/ambry.pdf http://dprg.cs.uiuc.edu/docs/SIGMOD2016-a/ambry.pdf
OhDagny 9y ago
Cassandra GC: check out scylla or the rocksandra projects.
bpicolo 9y ago
You can disk-back redis at various consistency levels if you so choose https://redis.io/topics/persistence https://redis.io/topics/persistence
nilkn 9y ago
Redis supports at least two forms of persistence: (1) periodic checkpoints; (2) writing an append-only log which can be periodically compacted to save space.
electrum 9y ago
> RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? https://myrocks.io/ https://myrocks.io/
petre 9y ago
They also released MyRocks, a RocksDB engine for MySQL which will is being intergrated in MariaDB 10.3 and Percona Server 5.7. I don't think it supports replication yet, but it's a start.
zilchers 9y ago
Cassandra has a host of other trade offs besides java (tombstones and data garbage collection come to mind).
- mspoof 9y ago
  Rocks Storage engine in Cassandra which claims to reduce the GC stalls https://engineering.instagram.com/open-sourcing-a-10x-reduction-in-apache-cassandra-tail-latency-d64f86b43589 https://engineering.instagram.com/open-sourcing-a-10x-reduct...

SanFranManDan 9y ago

I agree. However, I stumbled into the world of KV stores (like RocksDB, LMDB, LevelDB, etc..) last year, and what is most surprising is that they all stop in the same place. I understand that they should do one thing and one thing well, but it is still disappointing when you have to implement things like replication, sharding, and indexing yourself. There really aren't even that many DBMS that are KV (like redis) out there to handle it either. They are normally much more complicated (like adding SQL layer on top of it).

digikata 9y ago
I think we're just getting to the point where it may become more common to separate the simpler problem of a single node, non-distributed 'store' with some kv interface below, and then build more complex distributed algorithms in a layer or two above. You can see some of the larger monolithic codebases that had to start by having their own code all the way up and down the stack, but now are starting to experiment with backend store interfaces so you can trade-off some of the strengths and weaknesses of various local store performance areas. Along the same lines, a few newer codebases for distributed stores seem to be building with those delineations in mind. Another comment brought up Tidb/Tikv for example. Tikv iirc uses RocksDB as its local store.
- hyc_symas 9y ago
  "Just getting to the point" ? OpenLDAP has been architected this way for ~20 years. I think the same could be said for MySQL, as well as SQLServer (built on top of ESENT/JET). Large monolithic data stores are an obvious anti-pattern, reflects short-sighted design process.
  - digikata 9y ago
    Ha, you have a good point. I think my comment was directed more to "distributed cloud store" applications, much more so than databases. That begs the question - then what capabilities have those apps been focusing on that they didn't get from existing databases (or thought that they didn't get from existing databases)...

tracker1 9y ago

All the options you mention have varying features that give it advantages and disadvantages in various scenarios. What works well in on, won't in others, etc. That said, I was mostly interested on how this compares to etcd, since the use case seems pretty similar. How does Anna scale out (differing data centers) compared to it being a bad idea with etcd for example.

doorbumper 9y ago
I've just finished reading the paper. One difference is that Anna offers a weaker form of consistency than etcd. Etcd offers sequential consistency, and Anna offers casual consistency. I believe that this could eliminate many etcd use cases, since ordering for non-casual events many be different depending on the node you access. I'm not sure I fully understand etcd's durability guarantees. When an operation is completed, does that mean that the data is durable on a single machine, and then becomes durable on other machines at a later point? If so, it seems like Anna could offer the same durability guarantees. I think Anna's architecture could be more of a competitor to Cassandra, or DynamoDB as long as you only need casual consistency. The performance implications do seem pretty interesting.
- jaricftw 9y ago
  etcd does not provide key partitioning/sharding, it achieves linearizable consistency using replica state machine via Raft. Anna provides weaker consistency guarantee (causal or read committed), but has partitioning/sharding.

naasking 9y ago

> I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect it to be This paper literally shows this statement is false. In what other circumstances could you provide one and two orders of magnitude performance improvement, and people would just shrug and say, "meh, I really don't think we need improved performance".