5 ms·

This isn't a mutex, but the distributed equivalent of one. The storage service is the one who invalidates the lock on their side. The client won't detect its ow

This isn't a mutex, but the distributed equivalent of one. The storage service is the one who invalidates the lock on their side. The client won't detect its own issues without additional guarantees not given (supposedly) by Redlock.

neonbrain 2y ago
My understanding is that the dataflow user was talking about a notification which the server is supposed to receive from the OS in the case of a broken client connection. This notification is usually received, but cannot be guaranteed in a distributed environment.
dataflow 2y ago
I understand that. What I'm hung up on is, why does the storage system feel it is at liberty to just invalidate a lock and thus let someone else reacquire it without any sort of acknowledgment (either from the owner or from the communication systems connecting the owner to the outside world) that the owner will no longer rely on it? It just seems fundamentally wrong. The lock service just... doesn't have that liberty, as I see it.
- winwang 2y ago
  What if the rack goes down? But I think the author is saying a similar thing to you. The fenced token is essentially asserting that the client will no longer rely on the lock, even if it tries to. The difference is the service doesn't need any acknowledgement, no permission needed to simlly deny the client later.
  - dataflow 2y ago
    To be clear, my objection is to the premise, not to the offered solution. To your question, could you clarify what exactly you mean by the rack "going down"? This encompasses a lot of different scenarios, I'm not sure which one you're asking about. The obvious interpretation would break all the connections the program has to the outside world, thus preventing the problem by construction.
    - winwang 2y ago
      The rack could go down from the point of view of the storage service, but the machine/VM itself could be perfectly fine.
      - dataflow 2y ago
        In that scenario the machine would become aware that it can't reach the storage service either, no? In which case the host can terminate the program, or the network can break all the connections between them, or whatever. By default I would think that the lease shouldn't be broken until the network partition gets resolved, but I think the storage system could have a timeout for breaking the lease in that scenario if you really want, but then it would come with a time-based guarantee that the program isn't running anymore, no?
        winwang 2y ago
        Everything you're saying is plausibly possible in the absurdly large search space of all possible scenarios. The author's premise, however, is rooted in the specific scenario they lay out, with historical supporting examples which you can look into. Even then, the premise before all that was essentially: Redlock does not do what people might expect of a distributed lock. Btw I do have responses to your questions, but often times in these sorts of discussions, I find that there can always be an objection to an objection to ... etc. The "sense" (or flavor) in this case is that "we are taking a complex topic too lightly". In fact, I should probably continue reading the author's book (DDIA) at some point...
        dataflow 2y ago
        > The "sense" (or flavor) in this case is that "we are taking a complex topic too lightly". I get that -- and honestly, I'm not expecting a treatise on distributed consensus here. But what took me aback was that the blog post didn't even attempt to mention anything about the fact that the premise (at first glance) looks glaringly broken. If he'd even said 1 single sentence like "it's {difficult/infeasible/impossible} to design a client that will never continue execution past a timeout", it'd have been fine, and I would've happily moved along. But the way it is written right now, it reads a little bit like: "we design a ticking time bomb that we can't turn off; how can we make sure we don't forget to reset the timer every time?"... without bothering to say anything about why we should be digging ourselves into such a hole in the first place.

neonbrain 2y ago

My understanding is that the dataflow user was talking about a notification which the server is supposed to receive from the OS in the case of a broken client connection. This notification is usually received, but cannot be guaranteed in a distributed environment.

dataflow 2y ago

I understand that. What I'm hung up on is, why does the storage system feel it is at liberty to just invalidate a lock and thus let someone else reacquire it without any sort of acknowledgment (either from the owner or from the communication systems connecting the owner to the outside world) that the owner will no longer rely on it? It just seems fundamentally wrong. The lock service just... doesn't have that liberty, as I see it.

winwang 2y ago
What if the rack goes down? But I think the author is saying a similar thing to you. The fenced token is essentially asserting that the client will no longer rely on the lock, even if it tries to. The difference is the service doesn't need any acknowledgement, no permission needed to simlly deny the client later.
- dataflow 2y ago
  To be clear, my objection is to the premise, not to the offered solution. To your question, could you clarify what exactly you mean by the rack "going down"? This encompasses a lot of different scenarios, I'm not sure which one you're asking about. The obvious interpretation would break all the connections the program has to the outside world, thus preventing the problem by construction.
  - winwang 2y ago
    The rack could go down from the point of view of the storage service, but the machine/VM itself could be perfectly fine.
    - dataflow 2y ago
      In that scenario the machine would become aware that it can't reach the storage service either, no? In which case the host can terminate the program, or the network can break all the connections between them, or whatever. By default I would think that the lease shouldn't be broken until the network partition gets resolved, but I think the storage system could have a timeout for breaking the lease in that scenario if you really want, but then it would come with a time-based guarantee that the program isn't running anymore, no?
      - winwang 2y ago
        Everything you're saying is plausibly possible in the absurdly large search space of all possible scenarios. The author's premise, however, is rooted in the specific scenario they lay out, with historical supporting examples which you can look into. Even then, the premise before all that was essentially: Redlock does not do what people might expect of a distributed lock. Btw I do have responses to your questions, but often times in these sorts of discussions, I find that there can always be an objection to an objection to ... etc. The "sense" (or flavor) in this case is that "we are taking a complex topic too lightly". In fact, I should probably continue reading the author's book (DDIA) at some point...
        dataflow 2y ago
        > The "sense" (or flavor) in this case is that "we are taking a complex topic too lightly". I get that -- and honestly, I'm not expecting a treatise on distributed consensus here. But what took me aback was that the blog post didn't even attempt to mention anything about the fact that the premise (at first glance) looks glaringly broken. If he'd even said 1 single sentence like "it's {difficult/infeasible/impossible} to design a client that will never continue execution past a timeout", it'd have been fine, and I would've happily moved along. But the way it is written right now, it reads a little bit like: "we design a ticking time bomb that we can't turn off; how can we make sure we don't forget to reset the timer every time?"... without bothering to say anything about why we should be digging ourselves into such a hole in the first place.