5 ms·

Whatever is distributing the chunks is still a point of central coordination.

by kiitos 3y ago

Dylan16807 3y ago
Yes, and? If you can't manage minor levels of coordination because your database is on fire, the problem is that your database is on fire.
- kiitos 3y ago
  Fewer points of coordination is always better. In general you shouldn't need to make a roundtrip to produce an ID.
  - Dylan16807 3y ago
    > Fewer points of coordination is always better. The distributed database needs a coordination system anyway, so it's not an additional point. > In general you shouldn't need to make a roundtrip to produce an ID. Did you forget the context over the last week? We're already talking about reserving big chunks to remove the need to make a roundtrip to produce an ID. There would instead be something like one roundtrip per million IDs.
    - kiitos 3y ago
      > The distributed database needs a coordination system anyway, so it's not an additional point. Nope! Distributed databases do not necessarily need a "coordination system" in this sense. Most wide-scale distributed databases actually cannot rely on this kind of coordination. > Did you forget the context over the last week? We're already talking about reserving big chunks to remove the need to make a roundtrip to produce an ID. There would instead be something like one roundtrip per million IDs. OK, it's very clear that you're speaking from a context which is a very narrow subset of distributed systems as a whole. That's fine, just please understand your experience isn't broadly representative.
      - Dylan16807 3y ago
        > Nope! Distributed databases do not necessarily need a "coordination system" in this sense. Most wide-scale distributed databases actually cannot rely on this kind of coordination. I'm assuming a system that tracks nodes and checks for quorum(s), because if you let isolated servers be authoritative then your data integrity goes to hell. If you have that system, you can use it for low-bandwidth coordinated decisions like reserving blocks of ids. Am I wrong to think that most distributed databases have systems like that? > OK, it's very clear that you're speaking from a context which is a very narrow subset of distributed systems as a whole. That's fine, just please understand your experience isn't broadly representative. Sure, but the first thing you said in this conversation was "Whatever is distributing the chunks is still a point of central coordination." which is equally narrow, so I wasn't expecting you to suddenly broaden when I asked why that mattered.
        kiitos 3y ago
        > I'm assuming a system that tracks nodes and checks for quorum(s) Not sure why. > because if you let isolated servers be authoritative then your data integrity goes to hell Many AP systems maintain data integrity without central authorities or quorums for data. > Am I wrong to think that most distributed databases have systems like that? No, not wrong! Just that it's one class of distributed systems, among many.
        Dylan16807 3y ago
        Though if you're running AP then I sure hope you have a reconciliation system, and a good reconciliation system can handle that kind of ID conflict. (Maybe you still want to avoid it to speed that process up but that really gets into the weeds.)

Whatever is distributing the chunks is still a point of central coordination.

by kiitos 3y ago

Whatever is distributing the chunks is still a point of central coordination.

Yes, and? If you can't manage minor levels of coordination because your database is on fire, the problem is that your database is on fire.

kiitos 3y ago
Fewer points of coordination is always better. In general you shouldn't need to make a roundtrip to produce an ID.
- Dylan16807 3y ago
  > Fewer points of coordination is always better. The distributed database needs a coordination system anyway, so it's not an additional point. > In general you shouldn't need to make a roundtrip to produce an ID. Did you forget the context over the last week? We're already talking about reserving big chunks to remove the need to make a roundtrip to produce an ID. There would instead be something like one roundtrip per million IDs.
  - kiitos 3y ago
    > The distributed database needs a coordination system anyway, so it's not an additional point. Nope! Distributed databases do not necessarily need a "coordination system" in this sense. Most wide-scale distributed databases actually cannot rely on this kind of coordination. > Did you forget the context over the last week? We're already talking about reserving big chunks to remove the need to make a roundtrip to produce an ID. There would instead be something like one roundtrip per million IDs. OK, it's very clear that you're speaking from a context which is a very narrow subset of distributed systems as a whole. That's fine, just please understand your experience isn't broadly representative.
    - Dylan16807 3y ago
      > Nope! Distributed databases do not necessarily need a "coordination system" in this sense. Most wide-scale distributed databases actually cannot rely on this kind of coordination. I'm assuming a system that tracks nodes and checks for quorum(s), because if you let isolated servers be authoritative then your data integrity goes to hell. If you have that system, you can use it for low-bandwidth coordinated decisions like reserving blocks of ids. Am I wrong to think that most distributed databases have systems like that? > OK, it's very clear that you're speaking from a context which is a very narrow subset of distributed systems as a whole. That's fine, just please understand your experience isn't broadly representative. Sure, but the first thing you said in this conversation was "Whatever is distributing the chunks is still a point of central coordination." which is equally narrow, so I wasn't expecting you to suddenly broaden when I asked why that mattered.
      - kiitos 3y ago
        > I'm assuming a system that tracks nodes and checks for quorum(s) Not sure why. > because if you let isolated servers be authoritative then your data integrity goes to hell Many AP systems maintain data integrity without central authorities or quorums for data. > Am I wrong to think that most distributed databases have systems like that? No, not wrong! Just that it's one class of distributed systems, among many.
        Dylan16807 3y ago
        Though if you're running AP then I sure hope you have a reconciliation system, and a good reconciliation system can handle that kind of ID conflict. (Maybe you still want to avoid it to speed that process up but that really gets into the weeds.)