3 ms·

> The checkpoint system worked like this: every time we needed to perform a write, we would generate a UUID, store this UUID into a “checkpoint” for the current

> The checkpoint system worked like this: every time we needed to perform a write, we would generate a UUID, store this UUID into a “checkpoint” for the current flow.... >Logically, that was… fine. In practice, writes into the same database which previously required 5 IO round trips, now required almost double the number of trips for the extra checkpointing operations... So this sounds like trying to invent a form of optimistic locking, but it's definitely possible to do optimistic locking where no extra reads or writes are necessary unless there is a conflict. you have to write that value (whether timestamp or uuid) on every write, and then you do every write with a conditional "do this write as long as the lock value has not been changed." But I guess the store they were using didn't allow even a conditional write like that? Although I'm not sure how they managed to make this "checkpoint" system work without that feature either... this seems a bit confusing, it seems like a more typical optimistic locking system should have been possible using whatever primitives the checkpointing system used? Maybe not?

luuio 1y ago
The KV store had etag support for conditional writes. Etags are only useful to make sure the data didn't change underneath between your read and your write. Storing the checkpoints along with the mutation was for idempotency. If the checkpoint was in the document, that meant the mutation had succeeded and a retry should be no-op
- jrochkind1 1y ago
  Hm, I misunderstood, I thought the checkpoint system was also concurrency control to make sure nobody else had changed it from underneath you between read and write, since you had to read and write the whole (mega) document even though you only wanted to change a part (sub-document). Doesn't the KV provide idempotency on it's own -- so long as you're checking that no changes have happened between read and write, why wouldn't doing the same write twice produce an idempotent result? A change happenening between read and write seems the only reason that would be a problem. But clearly it's complicated and we don't have the whole picture as to business needs. Definitely sounds awful.
  - luuio 1y ago
    > why wouldn't doing the same write twice produce an idempotent result you can imagine this: ``` var thing = KVStore.Get<MyThing>(...); if (things.Checkpoints.Contains(myUuid) == false) { thing.Counter += 1; } KVStore.Update(thing); ``` having an etag doesn't help you with retries, where we expect that `thing` could be mutated by another flow between your retries.
    - jrochkind1 1y ago
      If the thing was mutated between your retries, then wasn't the etag changed by that mutation? So if you know the etag you started with, your conditional update on etag fails due to changed etag. So you fetch it again and start over. Which is the general optimistic locking algorithm. i may be missing something though?
      - luuio 1y ago
        let's say you have an object like this when you started: {count: 10, etag: 1}. then for some reason, something failed. when you retry and load the object, you get {count: 12, etag: 3}. how do you know if your previous attempt had successfully persisted or not, or if the updates to the object came from other processes/requests? you're mixing up conflict handling vs. idempotency
        deleted 1y ago
        [deleted]
        jrochkind1 1y ago
        Ah, the system is not capable of giving you a confirmation that your update succeeded, you don't really have a way to know without external "checkpoint" system? Anyway, okay, I get that I don't get it, and I get that it does sound terrible, agreed!
        luuio 1y ago
        Look up the two general problem on youtube. Unless the entire end to end operation is wrapped inside a giant transaction, no system in the world can give you the confirmation. Imagine this: you issued a write, a few things can happen: 1. The callsite crashed, maybe due to an out of memory issue or whatever. You don't know if it succeeded 2. The database returned an acknowledgement, but then the callsite crashed before storing the acknowlwedgement for the next step.

luuio 1y ago

The KV store had etag support for conditional writes. Etags are only useful to make sure the data didn't change underneath between your read and your write. Storing the checkpoints along with the mutation was for idempotency. If the checkpoint was in the document, that meant the mutation had succeeded and a retry should be no-op

jrochkind1 1y ago
Hm, I misunderstood, I thought the checkpoint system was also concurrency control to make sure nobody else had changed it from underneath you between read and write, since you had to read and write the whole (mega) document even though you only wanted to change a part (sub-document). Doesn't the KV provide idempotency on it's own -- so long as you're checking that no changes have happened between read and write, why wouldn't doing the same write twice produce an idempotent result? A change happenening between read and write seems the only reason that would be a problem. But clearly it's complicated and we don't have the whole picture as to business needs. Definitely sounds awful.
- luuio 1y ago
  > why wouldn't doing the same write twice produce an idempotent result you can imagine this: ``` var thing = KVStore.Get<MyThing>(...); if (things.Checkpoints.Contains(myUuid) == false) { thing.Counter += 1; } KVStore.Update(thing); ``` having an etag doesn't help you with retries, where we expect that `thing` could be mutated by another flow between your retries.
  - jrochkind1 1y ago
    If the thing was mutated between your retries, then wasn't the etag changed by that mutation? So if you know the etag you started with, your conditional update on etag fails due to changed etag. So you fetch it again and start over. Which is the general optimistic locking algorithm. i may be missing something though?
    - luuio 1y ago
      let's say you have an object like this when you started: {count: 10, etag: 1}. then for some reason, something failed. when you retry and load the object, you get {count: 12, etag: 3}. how do you know if your previous attempt had successfully persisted or not, or if the updates to the object came from other processes/requests? you're mixing up conflict handling vs. idempotency
      - deleted 1y ago
        [deleted]
      - jrochkind1 1y ago
        Ah, the system is not capable of giving you a confirmation that your update succeeded, you don't really have a way to know without external "checkpoint" system? Anyway, okay, I get that I don't get it, and I get that it does sound terrible, agreed!
        luuio 1y ago
        Look up the two general problem on youtube. Unless the entire end to end operation is wrapped inside a giant transaction, no system in the world can give you the confirmation. Imagine this: you issued a write, a few things can happen: 1. The callsite crashed, maybe due to an out of memory issue or whatever. You don't know if it succeeded 2. The database returned an acknowledgement, but then the callsite crashed before storing the acknowlwedgement for the next step.