7 ms·

There are two datastores I would really like that don't exist. 1) An efficient, high-performance distributed persistence store for arbitrarily large CRDT's. 2

There are two datastores I would really like that don't exist. 1) An efficient, high-performance distributed persistence store for arbitrarily large CRDT's. 2) A Kafka-like highly-available distributed binary log w/ cheap topics, that doesn't require external coordination, and doesn't lose acknowledged writes (which I'll happily give up any shape of linearization guarantee for).

endymi0n 11y ago
can't help with 1), but as for 2), maybe Kudu is an option for you? http://blog.rodeo.io/2016/01/24/kudu-as-a-more-flexible-kafka.html http://blog.rodeo.io/2016/01/24/kudu-as-a-more-flexible-kafk...
- im_down_w_otp 11y ago
  Uses Raft. Not highly-available enough unfortunately. Availability > Consistency for the use case that matters to me (basically a service message bus where all ACK'd writes will eventually be read by some consumer(s), but I don't care when or in what order, just that they all eventually get consumed, and where as long as any node in the cluster is up, it'll ACK a write).
  - Xorlev 11y ago
    > just that they all eventually get consumed Unless you have some sort of replication or consensus, this is still just best-effort delivery. For example, the message could be acknowledged & fsynced followed by drive death. At scale, this would happen. However, with asynchronous best-effort replication, broker death would be much less likely to lead to a loss of messages. In general, it's all about playing the odds. Acknowledged writes can still be lost in the best of systems if the entire DC catches fire. If you tune Kafka settings, you essentially get this behavior. You have to ensure ZK is up, but it will allow all topics to be published to so long as a single broker is alive.
    - im_down_w_otp 11y ago
      If I tune my Kafka settings I can't get this behavior. The closest I can get is allowing sloppy leader election, but that will definitely, definitely lose writes that have been ACK'd by a stale leader who comes back and has to become a follower. So I can get Kafka to basically accept all writes regardless of level of cluster degradation, but I can't keep it from dropping some of those writes silently onto the floor.
  - jasonwatkinspdx 11y ago
    > Uses Raft. Not highly-available enough unfortunately. Did you test this or is it just a general view? What sort of write rate are you looking for, per topic?
    - im_down_w_otp 11y ago
      Raft requires both a stable leader and a majority of replicas to be reachable in order to advance the consensus protocol. I'd like to be able to lose more than a majority of replicas and require no leaders because I don't care about consistency. I care that as long as even one node in the system is alive, it will accept writes, and those writes will eventually propagate and converge on the union set of all writes ever observed, without obliterating any ACK'd observed writes, as the system as a whole moves toward a less degraded state. I'd want topics themselves to be able to be basically "free" to create, and the write latency to remain stable over arbitrarily large datasets. Write rate isn't really much of a consideration since it's almost embarrassingly low (hundreds per second).
      - jasonwatkinspdx 11y ago
        > I don't care about consistency. Fair nuff, I asked because I was curious if you were assuming that eventual consistency was necessary for high availability. > Write rate isn't really much of a consideration since it's almost embarrassingly low (hundreds per second). Yup, looks like my concern above is the case. A single consensus replica set will do that no problem. I'd use raft if you're in a single geo location or epaxos otherwise (the epaxos source is a little early and needs eyeballs/production diversity testing, but the algorithm is absolutely solid). There's no reason to borrow all the troubles of CRDT's when it's not needed. Edit: Example from the epaxos paper, this is latency vs throughput for 3 replicas in one geo location: http://charap.co/wp-content/uploads/2015/10/latency_over_throughput.png http://charap.co/wp-content/uploads/2015/10/latency_over_thr...
        im_down_w_otp 11y ago
        What happens when I don't have a majority of Raft replicas alive?
chris_va 11y ago
(1) Spanner does exist, though not really available outside of Google
- atomic77 11y ago
  CockroachDB is probably the closest system to spanner that I've come across (and built in Go) https://github.com/cockroachdb/cockroach https://github.com/cockroachdb/cockroach
- im_down_w_otp 11y ago
  Yes, Spanner does exist. Spanner also does not satisfy what I said in #1.
  - QuercusMax 11y ago
    Can you elaborate on this?
    - im_down_w_otp 11y ago
      Well, for starters Google Spanner has almost nothing to do with CRDTs. For another it would be really weird for Spanner to expose something CRDT shaped since Spanner is a strongly-consistent distributed database which depends heavily on global order and read/write locks driven by meticulously coordinated multi-site global wall-clock time and consensus groups. In some ways it's almost the opposite of what I want... with exception of the fact that it's also distributed. I want extreme availability with lazy, weak coordination.
      - chris_va 11y ago
        Read/write locks... No, it's slightly more lazy in that that. It does timestamp coordination, but retrospectively.
        im_down_w_otp 11y ago
lobster_johnson 11y ago
Riak does CRDTs (not sure about arbitrarily large), though I never understood their reasoning for requiring that CRDTs be pre-registered in a schema — it's a major drawback in an otherwise schemaless K/V store.
- im_down_w_otp 11y ago
  But they're also required to be relatively small, because of how they're treated on disk and inside the system as opaque at the storage layer. As a result of this, and a smattering of other reasons, they're also not very fast.

endymi0n 11y ago

can't help with 1), but as for 2), maybe Kudu is an option for you? http://blog.rodeo.io/2016/01/24/kudu-as-a-more-flexible-kafka.html http://blog.rodeo.io/2016/01/24/kudu-as-a-more-flexible-kafk...

im_down_w_otp 11y ago
Uses Raft. Not highly-available enough unfortunately. Availability > Consistency for the use case that matters to me (basically a service message bus where all ACK'd writes will eventually be read by some consumer(s), but I don't care when or in what order, just that they all eventually get consumed, and where as long as any node in the cluster is up, it'll ACK a write).
- Xorlev 11y ago
  > just that they all eventually get consumed Unless you have some sort of replication or consensus, this is still just best-effort delivery. For example, the message could be acknowledged & fsynced followed by drive death. At scale, this would happen. However, with asynchronous best-effort replication, broker death would be much less likely to lead to a loss of messages. In general, it's all about playing the odds. Acknowledged writes can still be lost in the best of systems if the entire DC catches fire. If you tune Kafka settings, you essentially get this behavior. You have to ensure ZK is up, but it will allow all topics to be published to so long as a single broker is alive.
  - im_down_w_otp 11y ago
    If I tune my Kafka settings I can't get this behavior. The closest I can get is allowing sloppy leader election, but that will definitely, definitely lose writes that have been ACK'd by a stale leader who comes back and has to become a follower. So I can get Kafka to basically accept all writes regardless of level of cluster degradation, but I can't keep it from dropping some of those writes silently onto the floor.
- jasonwatkinspdx 11y ago
  > Uses Raft. Not highly-available enough unfortunately. Did you test this or is it just a general view? What sort of write rate are you looking for, per topic?
  - im_down_w_otp 11y ago
    Raft requires both a stable leader and a majority of replicas to be reachable in order to advance the consensus protocol. I'd like to be able to lose more than a majority of replicas and require no leaders because I don't care about consistency. I care that as long as even one node in the system is alive, it will accept writes, and those writes will eventually propagate and converge on the union set of all writes ever observed, without obliterating any ACK'd observed writes, as the system as a whole moves toward a less degraded state. I'd want topics themselves to be able to be basically "free" to create, and the write latency to remain stable over arbitrarily large datasets. Write rate isn't really much of a consideration since it's almost embarrassingly low (hundreds per second).
    - jasonwatkinspdx 11y ago
      > I don't care about consistency. Fair nuff, I asked because I was curious if you were assuming that eventual consistency was necessary for high availability. > Write rate isn't really much of a consideration since it's almost embarrassingly low (hundreds per second). Yup, looks like my concern above is the case. A single consensus replica set will do that no problem. I'd use raft if you're in a single geo location or epaxos otherwise (the epaxos source is a little early and needs eyeballs/production diversity testing, but the algorithm is absolutely solid). There's no reason to borrow all the troubles of CRDT's when it's not needed. Edit: Example from the epaxos paper, this is latency vs throughput for 3 replicas in one geo location: http://charap.co/wp-content/uploads/2015/10/latency_over_throughput.png http://charap.co/wp-content/uploads/2015/10/latency_over_thr...
      - im_down_w_otp 11y ago
        What happens when I don't have a majority of Raft replicas alive?

chris_va 11y ago

(1) Spanner does exist, though not really available outside of Google

atomic77 11y ago
CockroachDB is probably the closest system to spanner that I've come across (and built in Go) https://github.com/cockroachdb/cockroach https://github.com/cockroachdb/cockroach
im_down_w_otp 11y ago
Yes, Spanner does exist. Spanner also does not satisfy what I said in #1.
- QuercusMax 11y ago
  Can you elaborate on this?
  - im_down_w_otp 11y ago
    Well, for starters Google Spanner has almost nothing to do with CRDTs. For another it would be really weird for Spanner to expose something CRDT shaped since Spanner is a strongly-consistent distributed database which depends heavily on global order and read/write locks driven by meticulously coordinated multi-site global wall-clock time and consensus groups. In some ways it's almost the opposite of what I want... with exception of the fact that it's also distributed. I want extreme availability with lazy, weak coordination.
    - chris_va 11y ago
      Read/write locks... No, it's slightly more lazy in that that. It does timestamp coordination, but retrospectively.
      - im_down_w_otp 11y ago

lobster_johnson 11y ago

Riak does CRDTs (not sure about arbitrarily large), though I never understood their reasoning for requiring that CRDTs be pre-registered in a schema — it's a major drawback in an otherwise schemaless K/V store.

im_down_w_otp 11y ago
But they're also required to be relatively small, because of how they're treated on disk and inside the system as opaque at the storage layer. As a result of this, and a smattering of other reasons, they're also not very fast.