6 ms·

How to do distributed locking (2016)

244 points by yusufaytas 2y ago

galeaspablo 2y ago
Many engineers don’t truly care about the correctness issue, until it’s too late. Similar to security. Or they care but don’t bother checking whether what they’re doing is correct. For example, in my field, where microservices/actors/processes pass messages between each other over a network, I dare say >95% of implementations I see have edge cases where messages might be lost or processed out of order. But there isn’t an alignment of incentives that fixes this problem. Ie the payment structures for executives and engineers aren’t aligned with the best outcome for customers and shareholders.
- mrkeen 2y ago
  I think there's a bit of an alignment of incentives: the edge cases are tricky enough that your programmers probably need to handle a lot of support tickets, which isn't good for anyone. But I don't see anyway to convince yesterday's managers to give us time to build it right.
- sethammons 2y ago
  The path to fixing this requires first measuring and monitoring it, then establishing service level objectives that represent customer experience. Product and engineering teams have to agree on them. If the SLOs become violated, focus shifts towards system stability. Getting everyone onboard is hard and that is why good leadership is needed. When customers start to churn because bugs pop up and new features are slow or non existent, then the case is very easy to make quality part of the process. Mature leaders get ahead of that as early as possible.
  - galeaspablo 2y ago
    Good leadership is spot on! Agreed. The cynic part of me sees incentives that discourage mature leadership styles. Leaders tend to be impatient and think of this quarter’s OKRs as opposed to the business’ long term financial health. In other word the leaders of leaders use standard MBA prescribed incentive structures.
- secondcoming 2y ago
  > 95% of implementations I see have edge cases where messages might be lost or processed out of order. Eek. This sort of thing can end up with innocent people in jail, or dead. [0] https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal
  - noprocrasted 2y ago
    The problem (or the solution, depending on which side you're on) is that innocent people are in jail or dead. The people that knowingly allowed this to happen are still free and wealthy. So I'm not particularly sure this is a good example - if anything, it sets the opposite incentives, that even jailing people or driving them to suicide won't actually have any consequences for you.
- noprocrasted 2y ago
  > there isn’t an alignment of incentives that fixes this problem "Microservices" itself is often a symptom of this problem. Everyone and their dog wants to introduce a network boundary in between function calls for no good reason just so they can subsequently have endless busywork writing HTTP (or gRPC if you're lucky) servers, clients & JSON (de?)serializers for said function calls and try to reimplement things like distributed transactions across said network boundary and dealing with the inevitable "spooky action at a distance" that this will yield.
  - sethammons 2y ago
    I've worked with microservices at scale and it was fantastic. We couldn't break backwards compatibility with our API without a lot of coordination. Outside of that, you could deploy as frequently as needed and other services could update as needed to make use of new features. The monoliths I have worked in, very contrastingly, have had issues coordinating changes within the codebases, code crosses boundaries it should not and datastores get shared and coupled to (what should be) different domains leading to slow, inefficient code and ossified options for product changes.
  - shepherdjerred 2y ago
    If you're hand-writing clients/servers/serializers instead of generating them from schema definitions then you have more fundamental issues than using microservices.
    - sethammons 2y ago
      We hand wrote clients at my last microservice based gig. It was marginally slower than automated clients and we did run into a few cases of teams "waisting" their time writing their own clients; that was fixed by the authoring service team also authoring clients. It wasn't a big issue
- deleted 2y ago
  [deleted]
jmull 2y ago
This overcomplicates things... * If you have something like what the article calls a fencing token, you don't need any locks. * The token doesn't need to be monotonically increasing, just a passive unique value that both the client and storage have. Let's call it a version token. It could be monotonically increasing, but a generated UUID, which is typically easier, would work too. (Technically, it could even be a hash of all the data in the store, though that's probably not practical.) The logic becomes: (1) client retrieves the current version token from storage, along with any data it may want to modify. There's no external lock, though the storage needs to retrieve the data and version token atomically, ensuring the token is specifically for the version of the data retrieved. (2) client sends the version token back along with any changes. (3) Storage accepts the changes if the current token matches the one passed with the changes and creates a new version token (atomically, but still no external locks). Now, you can introduce locks for other reasons (hopefully goods ones... they seem to be misused a lot). Just pointing out they are/should be independent of storage integrity in a distributed system. (I don't even like the term lock, because they are temporary/unguaranteed. Lease or reservation might be a term that better conveys the meaning.)
- wh0knows 2y ago
  This neglects the first reason listed in the article for why you would use a lock. > Efficiency: Taking a lock saves you from unnecessarily doing the same work twice (e.g. some expensive computation). If the lock fails and two nodes end up doing the same piece of work, the result is a minor increase in cost (you end up paying 5 cents more to AWS than you otherwise would have) or a minor inconvenience (e.g. a user ends up getting the same email notification twice). I think multiple nodes doing the same work is actually much worse than what’s listed, as it would inhibit you from having any kind of scalable distributed processing.
  - karmakaze 2y ago
    As mentioned in the article, a non-100%-correct lock can be used for efficiency purposes. So basically use an imperfect locking mechanism for efficiency and a reliable one for correctness.
antirez 2y ago
I suggest reading the comment I left back then in this blog post comments section, and the reply I wrote in my blog. Btw, things to note in random order: 1. Check my comment under this blog post. The author had missed a fundamental point in how the algorithm works. Then he based the refusal of the algorithm on the remaining weaker points. 2. It is not true that you can't wait an approximately correct amount of time, with modern computers an APIs. GC pauses are bound and monotonic clocks work. These are acceptable assumptions. 3. To critique the auto release mechanism in-se, because you don't want to expose yourself to the fact that there is a potential race, is one thing. To critique the algorithm in front of its goals and its system model is another thing. 4. Over the years Redlock was used in a huge amount of use cases with success, because if you pick a timeout which is much larger than: A) the time to complete the task. B) the random pauses you can have in normal operating systems. Race conditions are very hard to trigger, and the other failures in the article were, AFAIK, never been observed. Of course if you have a super small timeout to auto release the lock, and the task may easily take this amount of time, you just committed a deisgn error, but that's not about Redlock.
- bluepizza 2y ago
  Could you provide links?
  - saikatsg 2y ago
    http://antirez.com/news/101 http://antirez.com/news/101
- computerfan494 2y ago
  To be honest I've long been puzzled by your response blog post. Maybe the following question can help achieve common ground: Would you use RedLock in a situation where the timeout is fairly short (1-2 seconds maybe), the work done usually takes ~90% of that timeout, and the work you do while holding a RedLock lock MUST NOT be done concurrently with another lock holder? I think the correct answer here is always "No" because the risk of the lease sometimes expiring before the client has finished its work is very high. You must alter your work to be idempotent because RedLock cannot guarantee mutual exclusion under all circumstances. Optimistic locking is a good way to implement this type of thing while the work done is idempotent.
eknkc 2y ago
I tend to use postgresql for distributed locking. As in, even if the job is not db related, I start a transaction and obtain an advisory lock which stays locked until the transaction is released. Either by the app itself or due to a crash or something. Felt pretty safe about it so far but I just realised I never check if the db connection is still ok. If this is a db related job and I need to touch the db, fine. Some query will fail on the connection and my job will fail anyway. Otherwise I might have already lost the lock and not aware of it. Without fencing tokens, atomic ops and such, I guess one needs a two stage commit on everything for absolute correctness?
- candiddevmike 2y ago
  One gotcha maybe with locks is they are connection specific AFAIK, and in most libraries you're using a pool typically. So you need to have a specific connection for locks, and ensure you're using that connection when doing periodic lock tests.
  - Quekid5 2y ago
    Why would locks be connection-specific? ... considering that only one operation can be in flight at a time on a single connection. (Usually, at least.)
    - joatmon-snoo 2y ago
      Different DBs implement locks differently. Postgres allows obtaining advisory locks at either the session _or_ transaction level. If it's session-level, then you have, ergo, a connection-level lock. https://www.postgresql.org/docs/current/explicit-locking.html https://www.postgresql.org/docs/current/explicit-locking.htm...
  - skrause 2y ago
    PostgreSQL has pg_advisory_xact_lock which releases the lock automatically when the transaction is over.
    - 2y ago
jroseattle 2y ago
We reviewed Redis back in 2018 as a potential solution for our use case. In the end, we opted for a less sexy solution (not Redis) that never failed us, no joke. Our use case: handing out a ticket (something with an identifier) from a finite set of tickets from a campaign. It's something akin to Ticketmaster allocating seats in a venue for a concert. Our operation was as you might expect: provide a ticket to a request if one is available, assign some metadata from the request to the allocated ticket, and remove it from consideration for future client requests. We had failed campaigns in the past (over-allocation, under-allocation, duplicate allocation, etc.) so our concern was accuracy. Clients would connect and request a ticket; we wanted to exclusively distribute only the set of tickets available from the pool. If the number of client requests exceeded the number of tickets, the system should protect for that. We tried Redis, including the naive implementation of getting the lock, checking the lock, doing our thing, releasing the lock. It was ok, but administrative overhead was a lot for us at the time. I'm glad we didn't go that route, though. We ultimately settled on...Postgres. Our "distributed lock" was just a composite UPDATE statement using some Postgres-specific features. We effectively turned requests into a SET operation, where the database would return either a record that indicated the request was successful, or something that indicated it failed. ACID transactions for the win! With accuracy solved, we next looked at scale/performance. We didn't need to support millions of requests/sec, but we did have some spikiness thresholds. We were able to optimize read/write db instances within our cluster, and strategically load larger/higher-demand campaigns to allocated systems. We continued to improve on optimization over two years, but not once did we ever have a campaign with ticket distribution failures. Note: I am not an expert of any kind in distributed-lock technology. I'm just someone who did their homework, focused on the problem to be solved, and found a solution after trying a few things.
- etcd 2y ago
  I guess this is embarassingly parralelizable in that you can shard by concert to different instances. Might even be a job for that newfangled cloudflare sqlite thing.
- wwarner 2y ago
  This is the best way, and actually the only sensible way to approach the problem. I first read about it here https://code.flickr.net/2010/02/08/ticket-servers-distributed-unique-primary-keys-on-the-cheap/ https://code.flickr.net/2010/02/08/ticket-servers-distribute...
hoppp 2y ago
I did distributed locking with Deno, and Deno KV hosted by Deno Deploy. Its using foundationdb, a distributed db. The deno instances running on local devices all connect to the same Deno KV to acquire the lock. But using postgres, a select for update also works, the database is not distributed tho.
egcodes 2y ago
Once I wrote a dist. lock blog using this resource. Here it is: https://medium.com/sahibinden-technology/an-easy-integration-of-distributed-lock-4b19a704ce49 https://medium.com/sahibinden-technology/an-easy-integration...
anonzzzies 2y ago
I am updating my low level and algo knowledge; what are good books about this (I have the one written by the author). I am looking to build something for fun, but everything is either a toy or very complicated.
- cosmicradiance 2y ago
  System Design Interview I and II - Alex Xu. Take one of the topics and do it practically.
dataflow 2y ago
> The lock has a timeout (i.e. it is a lease), which is always a good idea (otherwise a crashed client could end up holding a lock forever and never releasing it). However, if the GC pause lasts longer than the lease expiry period, and the client doesn’t realise that it has expired, it may go ahead and make some unsafe change. Hold on, this sounds absurd to me: First, if your client crashes, then you don't need a timed lease on the lock to detect this in the first place. The lock would get released by the OS or supervisor, whether there are any timeouts or not. If both of those crash too, then the connection would eventually break, and the network system should then detect that (via network resets or timeouts, lack of heartbeats, etc.) and then invalidate all your connections before releasing any locks. Second, if the problem becomes that your client is buggy and thus holds the lock too long without crashing, then shouldn't some kind of supervisor detect that and then kill the client (e.g., by the OS terminating the process) before releasing the lock for everybody else? Third, if you are going to have locks with timeouts to deal with corner cases you can't handle like the above, shouldn't they notify the actual program somehow (e.g., by throwing an exception, raising a signal, terminating it, etc.) instead of letting it happily continue execution? And shouldn't those cases wait for some kind of verification that the program was notified before releasing the lock? The whole notion that timeouts should somehow permit the program execution to continue ordinary control flow sounds like the root cause of the problem, and nobody is even batting an eye at it? Is there an obvious reason why this makes sense? I feel I must be missing something here... what am I missing?
- neonbrain 2y ago
  The assumption that your server will always receive RST or FIN from your client is incorrect. There are some cases when these packets are being dropped, and your server will stay with an open connection while the client on the remote machine is already dead. P.S. BTW, it's not me who downvoted you
  - dataflow 2y ago
    I made no such assumption this will always happen though? That's why the comment was so much longer than just "isn't TCP RST enough?"... I listed a ton of ways to deal with this that didn't involve letting the program continue happily on its path.
jojolatulipe 2y ago
At work we use Temporal and ended up using a dedicated workflow and signals to do distributed locking. Working well so far and the implementation is rather simple, relying on Temporal’s facilities to do the distributed parts of the lock.
- robertlagrant 2y ago
  I'm keen to use Temporal, but I've heard it can be flaky. In your experience has it worked well?
  - calmoo 2y ago
    Rock solid in my experience and kind of a game changer. I’m surprised it’s not more widespread in large orgs.
    - Icathian 2y ago
      We use it a ton at my shop for internal things like release rollouts. Fairly big tech company, and same experience. It's an excellent product.
- wslh 2y ago
  I just discovered Temporal, and I have to say thank you! From what I've seen so far, it seems like the holy grail for workflows, offering very clear high-level task management over complex infrastructure. Is Temporal unique in this space, or are there other alternatives of similar caliber? Given that it was spun off from Uber and is used by top vendors, it sounds like it’s been thoroughly battle-tested.
  - refset 2y ago
    DBOS [0] is outwardly similar although it's much younger. IIUC, internally DBOS is able to be more efficient and support lower latencies than Temporal because of the way it can push work down into Postgres stored procedures. [0] https://www.dbos.dev/ https://www.dbos.dev/

244 points by yusufaytas 2y ago

Many engineers don’t truly care about the correctness issue, until it’s too late. Similar to security. Or they care but don’t bother checking whether what they’re doing is correct. For example, in my field, where microservices/actors/processes pass messages between each other over a network, I dare say >95% of implementations I see have edge cases where messages might be lost or processed out of order. But there isn’t an alignment of incentives that fixes this problem. Ie the payment structures for executives and engineers aren’t aligned with the best outcome for customers and shareholders.

mrkeen 2y ago
I think there's a bit of an alignment of incentives: the edge cases are tricky enough that your programmers probably need to handle a lot of support tickets, which isn't good for anyone. But I don't see anyway to convince yesterday's managers to give us time to build it right.
sethammons 2y ago
The path to fixing this requires first measuring and monitoring it, then establishing service level objectives that represent customer experience. Product and engineering teams have to agree on them. If the SLOs become violated, focus shifts towards system stability. Getting everyone onboard is hard and that is why good leadership is needed. When customers start to churn because bugs pop up and new features are slow or non existent, then the case is very easy to make quality part of the process. Mature leaders get ahead of that as early as possible.
- galeaspablo 2y ago
  Good leadership is spot on! Agreed. The cynic part of me sees incentives that discourage mature leadership styles. Leaders tend to be impatient and think of this quarter’s OKRs as opposed to the business’ long term financial health. In other word the leaders of leaders use standard MBA prescribed incentive structures.
secondcoming 2y ago
> 95% of implementations I see have edge cases where messages might be lost or processed out of order. Eek. This sort of thing can end up with innocent people in jail, or dead. [0] https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal
- noprocrasted 2y ago
  The problem (or the solution, depending on which side you're on) is that innocent people are in jail or dead. The people that knowingly allowed this to happen are still free and wealthy. So I'm not particularly sure this is a good example - if anything, it sets the opposite incentives, that even jailing people or driving them to suicide won't actually have any consequences for you.
noprocrasted 2y ago
> there isn’t an alignment of incentives that fixes this problem "Microservices" itself is often a symptom of this problem. Everyone and their dog wants to introduce a network boundary in between function calls for no good reason just so they can subsequently have endless busywork writing HTTP (or gRPC if you're lucky) servers, clients & JSON (de?)serializers for said function calls and try to reimplement things like distributed transactions across said network boundary and dealing with the inevitable "spooky action at a distance" that this will yield.
- sethammons 2y ago
  I've worked with microservices at scale and it was fantastic. We couldn't break backwards compatibility with our API without a lot of coordination. Outside of that, you could deploy as frequently as needed and other services could update as needed to make use of new features. The monoliths I have worked in, very contrastingly, have had issues coordinating changes within the codebases, code crosses boundaries it should not and datastores get shared and coupled to (what should be) different domains leading to slow, inefficient code and ossified options for product changes.
- shepherdjerred 2y ago
  If you're hand-writing clients/servers/serializers instead of generating them from schema definitions then you have more fundamental issues than using microservices.
  - sethammons 2y ago
    We hand wrote clients at my last microservice based gig. It was marginally slower than automated clients and we did run into a few cases of teams "waisting" their time writing their own clients; that was fixed by the authoring service team also authoring clients. It wasn't a big issue
deleted 2y ago
[deleted]

jmull 2y ago

This overcomplicates things... * If you have something like what the article calls a fencing token, you don't need any locks. * The token doesn't need to be monotonically increasing, just a passive unique value that both the client and storage have. Let's call it a version token. It could be monotonically increasing, but a generated UUID, which is typically easier, would work too. (Technically, it could even be a hash of all the data in the store, though that's probably not practical.) The logic becomes: (1) client retrieves the current version token from storage, along with any data it may want to modify. There's no external lock, though the storage needs to retrieve the data and version token atomically, ensuring the token is specifically for the version of the data retrieved. (2) client sends the version token back along with any changes. (3) Storage accepts the changes if the current token matches the one passed with the changes and creates a new version token (atomically, but still no external locks). Now, you can introduce locks for other reasons (hopefully goods ones... they seem to be misused a lot). Just pointing out they are/should be independent of storage integrity in a distributed system. (I don't even like the term lock, because they are temporary/unguaranteed. Lease or reservation might be a term that better conveys the meaning.)

wh0knows 2y ago
This neglects the first reason listed in the article for why you would use a lock. > Efficiency: Taking a lock saves you from unnecessarily doing the same work twice (e.g. some expensive computation). If the lock fails and two nodes end up doing the same piece of work, the result is a minor increase in cost (you end up paying 5 cents more to AWS than you otherwise would have) or a minor inconvenience (e.g. a user ends up getting the same email notification twice). I think multiple nodes doing the same work is actually much worse than what’s listed, as it would inhibit you from having any kind of scalable distributed processing.
- karmakaze 2y ago
  As mentioned in the article, a non-100%-correct lock can be used for efficiency purposes. So basically use an imperfect locking mechanism for efficiency and a reliable one for correctness.

antirez 2y ago

I suggest reading the comment I left back then in this blog post comments section, and the reply I wrote in my blog. Btw, things to note in random order: 1. Check my comment under this blog post. The author had missed a fundamental point in how the algorithm works. Then he based the refusal of the algorithm on the remaining weaker points. 2. It is not true that you can't wait an approximately correct amount of time, with modern computers an APIs. GC pauses are bound and monotonic clocks work. These are acceptable assumptions. 3. To critique the auto release mechanism in-se, because you don't want to expose yourself to the fact that there is a potential race, is one thing. To critique the algorithm in front of its goals and its system model is another thing. 4. Over the years Redlock was used in a huge amount of use cases with success, because if you pick a timeout which is much larger than: A) the time to complete the task. B) the random pauses you can have in normal operating systems. Race conditions are very hard to trigger, and the other failures in the article were, AFAIK, never been observed. Of course if you have a super small timeout to auto release the lock, and the task may easily take this amount of time, you just committed a deisgn error, but that's not about Redlock.

bluepizza 2y ago
Could you provide links?
- saikatsg 2y ago
  http://antirez.com/news/101 http://antirez.com/news/101
computerfan494 2y ago
To be honest I've long been puzzled by your response blog post. Maybe the following question can help achieve common ground: Would you use RedLock in a situation where the timeout is fairly short (1-2 seconds maybe), the work done usually takes ~90% of that timeout, and the work you do while holding a RedLock lock MUST NOT be done concurrently with another lock holder? I think the correct answer here is always "No" because the risk of the lease sometimes expiring before the client has finished its work is very high. You must alter your work to be idempotent because RedLock cannot guarantee mutual exclusion under all circumstances. Optimistic locking is a good way to implement this type of thing while the work done is idempotent.

eknkc 2y ago

I tend to use postgresql for distributed locking. As in, even if the job is not db related, I start a transaction and obtain an advisory lock which stays locked until the transaction is released. Either by the app itself or due to a crash or something. Felt pretty safe about it so far but I just realised I never check if the db connection is still ok. If this is a db related job and I need to touch the db, fine. Some query will fail on the connection and my job will fail anyway. Otherwise I might have already lost the lock and not aware of it. Without fencing tokens, atomic ops and such, I guess one needs a two stage commit on everything for absolute correctness?

candiddevmike 2y ago
One gotcha maybe with locks is they are connection specific AFAIK, and in most libraries you're using a pool typically. So you need to have a specific connection for locks, and ensure you're using that connection when doing periodic lock tests.
- Quekid5 2y ago
  Why would locks be connection-specific? ... considering that only one operation can be in flight at a time on a single connection. (Usually, at least.)
  - joatmon-snoo 2y ago
    Different DBs implement locks differently. Postgres allows obtaining advisory locks at either the session _or_ transaction level. If it's session-level, then you have, ergo, a connection-level lock. https://www.postgresql.org/docs/current/explicit-locking.html https://www.postgresql.org/docs/current/explicit-locking.htm...
- skrause 2y ago
  PostgreSQL has pg_advisory_xact_lock which releases the lock automatically when the transaction is over.
  - 2y ago

jroseattle 2y ago

We reviewed Redis back in 2018 as a potential solution for our use case. In the end, we opted for a less sexy solution (not Redis) that never failed us, no joke. Our use case: handing out a ticket (something with an identifier) from a finite set of tickets from a campaign. It's something akin to Ticketmaster allocating seats in a venue for a concert. Our operation was as you might expect: provide a ticket to a request if one is available, assign some metadata from the request to the allocated ticket, and remove it from consideration for future client requests. We had failed campaigns in the past (over-allocation, under-allocation, duplicate allocation, etc.) so our concern was accuracy. Clients would connect and request a ticket; we wanted to exclusively distribute only the set of tickets available from the pool. If the number of client requests exceeded the number of tickets, the system should protect for that. We tried Redis, including the naive implementation of getting the lock, checking the lock, doing our thing, releasing the lock. It was ok, but administrative overhead was a lot for us at the time. I'm glad we didn't go that route, though. We ultimately settled on...Postgres. Our "distributed lock" was just a composite UPDATE statement using some Postgres-specific features. We effectively turned requests into a SET operation, where the database would return either a record that indicated the request was successful, or something that indicated it failed. ACID transactions for the win! With accuracy solved, we next looked at scale/performance. We didn't need to support millions of requests/sec, but we did have some spikiness thresholds. We were able to optimize read/write db instances within our cluster, and strategically load larger/higher-demand campaigns to allocated systems. We continued to improve on optimization over two years, but not once did we ever have a campaign with ticket distribution failures. Note: I am not an expert of any kind in distributed-lock technology. I'm just someone who did their homework, focused on the problem to be solved, and found a solution after trying a few things.

etcd 2y ago
I guess this is embarassingly parralelizable in that you can shard by concert to different instances. Might even be a job for that newfangled cloudflare sqlite thing.
wwarner 2y ago
This is the best way, and actually the only sensible way to approach the problem. I first read about it here https://code.flickr.net/2010/02/08/ticket-servers-distributed-unique-primary-keys-on-the-cheap/ https://code.flickr.net/2010/02/08/ticket-servers-distribute...

hoppp 2y ago

I did distributed locking with Deno, and Deno KV hosted by Deno Deploy. Its using foundationdb, a distributed db. The deno instances running on local devices all connect to the same Deno KV to acquire the lock. But using postgres, a select for update also works, the database is not distributed tho.

egcodes 2y ago

Once I wrote a dist. lock blog using this resource. Here it is: https://medium.com/sahibinden-technology/an-easy-integration-of-distributed-lock-4b19a704ce49 https://medium.com/sahibinden-technology/an-easy-integration...

anonzzzies 2y ago

I am updating my low level and algo knowledge; what are good books about this (I have the one written by the author). I am looking to build something for fun, but everything is either a toy or very complicated.

cosmicradiance 2y ago
System Design Interview I and II - Alex Xu. Take one of the topics and do it practically.

dataflow 2y ago

> The lock has a timeout (i.e. it is a lease), which is always a good idea (otherwise a crashed client could end up holding a lock forever and never releasing it). However, if the GC pause lasts longer than the lease expiry period, and the client doesn’t realise that it has expired, it may go ahead and make some unsafe change. Hold on, this sounds absurd to me: First, if your client crashes, then you don't need a timed lease on the lock to detect this in the first place. The lock would get released by the OS or supervisor, whether there are any timeouts or not. If both of those crash too, then the connection would eventually break, and the network system should then detect that (via network resets or timeouts, lack of heartbeats, etc.) and then invalidate all your connections before releasing any locks. Second, if the problem becomes that your client is buggy and thus holds the lock too long without crashing, then shouldn't some kind of supervisor detect that and then kill the client (e.g., by the OS terminating the process) before releasing the lock for everybody else? Third, if you are going to have locks with timeouts to deal with corner cases you can't handle like the above, shouldn't they notify the actual program somehow (e.g., by throwing an exception, raising a signal, terminating it, etc.) instead of letting it happily continue execution? And shouldn't those cases wait for some kind of verification that the program was notified before releasing the lock? The whole notion that timeouts should somehow permit the program execution to continue ordinary control flow sounds like the root cause of the problem, and nobody is even batting an eye at it? Is there an obvious reason why this makes sense? I feel I must be missing something here... what am I missing?

neonbrain 2y ago
The assumption that your server will always receive RST or FIN from your client is incorrect. There are some cases when these packets are being dropped, and your server will stay with an open connection while the client on the remote machine is already dead. P.S. BTW, it's not me who downvoted you
- dataflow 2y ago
  I made no such assumption this will always happen though? That's why the comment was so much longer than just "isn't TCP RST enough?"... I listed a ton of ways to deal with this that didn't involve letting the program continue happily on its path.

jojolatulipe 2y ago

At work we use Temporal and ended up using a dedicated workflow and signals to do distributed locking. Working well so far and the implementation is rather simple, relying on Temporal’s facilities to do the distributed parts of the lock.

robertlagrant 2y ago
I'm keen to use Temporal, but I've heard it can be flaky. In your experience has it worked well?
- calmoo 2y ago
  Rock solid in my experience and kind of a game changer. I’m surprised it’s not more widespread in large orgs.
  - Icathian 2y ago
    We use it a ton at my shop for internal things like release rollouts. Fairly big tech company, and same experience. It's an excellent product.
wslh 2y ago
I just discovered Temporal, and I have to say thank you! From what I've seen so far, it seems like the holy grail for workflows, offering very clear high-level task management over complex infrastructure. Is Temporal unique in this space, or are there other alternatives of similar caliber? Given that it was spun off from Uber and is used by top vendors, it sounds like it’s been thoroughly battle-tested.
- refset 2y ago
  DBOS [0] is outwardly similar although it's much younger. IIUC, internally DBOS is able to be more efficient and support lower latencies than Temporal because of the way it can push work down into Postgres stored procedures. [0] https://www.dbos.dev/ https://www.dbos.dev/