4 ms·

So the website mentions that a Crate cluster is a share-nothing system; That each node can act independently. This would mean that the data is limited to one ma

So the website mentions that a Crate cluster is a share-nothing system; That each node can act independently. This would mean that the data is limited to one machines storage size? Doesn't really fit in with the claim of a distributed database unless I'm missing something.

jodok 12y ago
No, it can hold as much data as the sum of all the nodes in the cluster have (also depending on the amount of replicas you configure). Some background: All data is being sharded by default. These shards are distributed evenly between the nodes. Nodes can hold multiple shards. Furthermore shards can have replicas, being hold by other nodes. Memory is being utilized to hold indizes in RAM and to cache values that are being needed frequently for e.g. aggregations ("field caches").
spanktar 12y ago
By shared-nothing, we mean that no node is special, no node is indispensable. Unlike other popular NoSQL databases, we don't have different classes of nodes. The persistent data is distributed across the cluster, and nodes communicate amongst each other about which data are where. http://en.wikipedia.org/wiki/Shared_nothing_architecture http://en.wikipedia.org/wiki/Shared_nothing_architecture
- fizx 12y ago
  Given that this is based on Elasticsearch, there will be a master node, which should be chosen explicitly in configuration to be away from the data nodes, because ES responds poorly to garbage collection on the master node. So you want to keep the load down on your master. So you can run in a sort-of-shared-nothing configuration, but its not recommended.
  - spanktar 12y ago
    True. There is an elected master node and you can configure a cluster to have master-only (non-data) nodes, data-only nodes (non-master eligible), or both. Even still, and even if you choose to set up a cluster this way, every node can perform the same functions, it's more a matter of what responsibility a node has. And even though there is a master node, that master can disappear without consequence, another will be elected in its stead. There is nothing really "special" about a master node.

jodok 12y ago

No, it can hold as much data as the sum of all the nodes in the cluster have (also depending on the amount of replicas you configure). Some background: All data is being sharded by default. These shards are distributed evenly between the nodes. Nodes can hold multiple shards. Furthermore shards can have replicas, being hold by other nodes. Memory is being utilized to hold indizes in RAM and to cache values that are being needed frequently for e.g. aggregations ("field caches").

spanktar 12y ago

By shared-nothing, we mean that no node is special, no node is indispensable. Unlike other popular NoSQL databases, we don't have different classes of nodes. The persistent data is distributed across the cluster, and nodes communicate amongst each other about which data are where. http://en.wikipedia.org/wiki/Shared_nothing_architecture http://en.wikipedia.org/wiki/Shared_nothing_architecture

fizx 12y ago
Given that this is based on Elasticsearch, there will be a master node, which should be chosen explicitly in configuration to be away from the data nodes, because ES responds poorly to garbage collection on the master node. So you want to keep the load down on your master. So you can run in a sort-of-shared-nothing configuration, but its not recommended.
- spanktar 12y ago
  True. There is an elected master node and you can configure a cluster to have master-only (non-data) nodes, data-only nodes (non-master eligible), or both. Even still, and even if you choose to set up a cluster this way, every node can perform the same functions, it's more a matter of what responsibility a node has. And even though there is a master node, that master can disappear without consequence, another will be elected in its stead. There is nothing really "special" about a master node.