3 ms·

I'm wondering chow CockroachDB will work in a distributed mesh network. I've never seen someone talk about it but how would this work out?

tylerkahn 10y ago
Raft (the underlying consensus protocol) is agnostic to the underlying network topology/implementation.
ansible 10y ago
What do you mean exactly? It is designed to run across multiple nodes, preferably out of multiple datacenters that don't have common points of failure (network connectivity, power). But it is designed to run on systems that you control, as opposed to some other peer-to-peer systems.
- gravypod 10y ago
  It's going to be P2P nodes I control. The idea is to do IOT logging of data and put it into a DB to query everything basically. That's the long and short of it at least.
  - ansible 10y ago
    I don't think CockroachDB will be a good fit for your application. The nodes expect constant network connectivity, they're talking to each other every few milliseconds. You want a very fast and reliable network. For logging, some sort of store and forward architecture may be better. I don't think the level of consistency provided by CockroachDB would be necessary.
teraflop 10y ago
Well, the basic design of Cockroach (assuming no bugs) prioritizes consistency over availability, which means that your data should be safe even if the network is misbehaves arbitrarily. So the really interesting question is, how much availability can you expect? "Mesh network" implies that the network should reconfigure itself if individual nodes or links fail. That's fine, but it will necessarily take some time to detect a failed connection, which means temporary unavailability while the database performs a leader election. CockroachDB is sharded, so if any single shard loses a majority of its nodes or becomes too badly partitioned, that shard becomes unavailabile until the network is restored. There's a tradeoff here -- if you're willing to accept slower performance, you can increase the number of replicas per shard. (It's also interesting to think about what happens under heavy load. If the Cockroach daemons are handling requests as fast as they can, then whatever daemon is responsible for the mesh routing decision-making could become starved for CPU, leading to high packet loss or intermittent network partitions.)
- gravypod 10y ago
  Ideally the only thing running on the nodes is a set of programs logging data. The actual queries should come from outside the network. Is it possible to get CDB to exfiltrate it's data into a single node or another cluster of nodes?
erichocean 10y ago
> how would this work out? Poorly.

tylerkahn 10y ago

Raft (the underlying consensus protocol) is agnostic to the underlying network topology/implementation.

ansible 10y ago

What do you mean exactly? It is designed to run across multiple nodes, preferably out of multiple datacenters that don't have common points of failure (network connectivity, power). But it is designed to run on systems that you control, as opposed to some other peer-to-peer systems.

gravypod 10y ago
It's going to be P2P nodes I control. The idea is to do IOT logging of data and put it into a DB to query everything basically. That's the long and short of it at least.
- ansible 10y ago
  I don't think CockroachDB will be a good fit for your application. The nodes expect constant network connectivity, they're talking to each other every few milliseconds. You want a very fast and reliable network. For logging, some sort of store and forward architecture may be better. I don't think the level of consistency provided by CockroachDB would be necessary.

teraflop 10y ago

Well, the basic design of Cockroach (assuming no bugs) prioritizes consistency over availability, which means that your data should be safe even if the network is misbehaves arbitrarily. So the really interesting question is, how much availability can you expect? "Mesh network" implies that the network should reconfigure itself if individual nodes or links fail. That's fine, but it will necessarily take some time to detect a failed connection, which means temporary unavailability while the database performs a leader election. CockroachDB is sharded, so if any single shard loses a majority of its nodes or becomes too badly partitioned, that shard becomes unavailabile until the network is restored. There's a tradeoff here -- if you're willing to accept slower performance, you can increase the number of replicas per shard. (It's also interesting to think about what happens under heavy load. If the Cockroach daemons are handling requests as fast as they can, then whatever daemon is responsible for the mesh routing decision-making could become starved for CPU, leading to high packet loss or intermittent network partitions.)

gravypod 10y ago
Ideally the only thing running on the nodes is a set of programs logging data. The actual queries should come from outside the network. Is it possible to get CDB to exfiltrate it's data into a single node or another cluster of nodes?

erichocean 10y ago

> how would this work out? Poorly.