5 ms·

> Every server in the cluster keeps an in memory copy of this cluster metadata. Each will periodically refresh the entire metadata set to pick up any changes.

> Every server in the cluster keeps an in memory copy of this cluster metadata. Each will periodically refresh the entire metadata set to pick up any changes. How does this avoid nodes doing stale reads from the inmemory copy resulting in each node having a slightly different out-of-date view of the cluster?

lobster_johnson 11y ago
I believe Raft is supposed to handle this; every modification is a log entry, and every recipient has to ack the log entry, a bit like two-phased commit.
- dcb18 11y ago
  This is incorrect, only reads from the current Raft master are guaranteed to not be stale. In the case of InfluxDB, I think caching is safe because the shard metadata is immutable.
  - schmichael 11y ago
    Where are you seeing that cluster metadata is immutable? I don't even know how that would work. Surely nodes, databases, shards, users, permissions, etc. all can change?
    - dcb18 11y ago
      Yep, you're right.
- schmichael 11y ago
  But the section I quoted said that local metadata queries go to the cache which is only updated "periodically."
pauldix 11y ago
They could have an out of date view of the cluster. Like which databases exist, or which servers exist. However, those cases are fine. If a database was created and a server doesn't have a copy of it. When a write comes in it'll be a cache miss so it'll hit the Raft system to get the info. In the case of a new server joining the cluster, it won't get new data assigned until a shard group gets created and a shard gets assigned to it. When a write comes in if a node doesn't have a shard group for that time range, it'll all out to the Raft system to get the information. So yes, it's possible for some servers to have a stale view of this cluster metadata, but we work around it by having them request the information on demand and periodically refresh the entire thing.
- schmichael 11y ago
  Thanks for the more detailed explanation! Really looking forward to further posts -- not to mention 0.9!

lobster_johnson 11y ago

I believe Raft is supposed to handle this; every modification is a log entry, and every recipient has to ack the log entry, a bit like two-phased commit.

dcb18 11y ago
This is incorrect, only reads from the current Raft master are guaranteed to not be stale. In the case of InfluxDB, I think caching is safe because the shard metadata is immutable.
- schmichael 11y ago
  Where are you seeing that cluster metadata is immutable? I don't even know how that would work. Surely nodes, databases, shards, users, permissions, etc. all can change?
  - dcb18 11y ago
    Yep, you're right.
schmichael 11y ago
But the section I quoted said that local metadata queries go to the cache which is only updated "periodically."

pauldix 11y ago

They could have an out of date view of the cluster. Like which databases exist, or which servers exist. However, those cases are fine. If a database was created and a server doesn't have a copy of it. When a write comes in it'll be a cache miss so it'll hit the Raft system to get the info. In the case of a new server joining the cluster, it won't get new data assigned until a shard group gets created and a shard gets assigned to it. When a write comes in if a node doesn't have a shard group for that time range, it'll all out to the Raft system to get the information. So yes, it's possible for some servers to have a stale view of this cluster metadata, but we work around it by having them request the information on demand and periodically refresh the entire thing.

schmichael 11y ago
Thanks for the more detailed explanation! Really looking forward to further posts -- not to mention 0.9!