5 ms·
Show HN: Lockable – sync locks for distributed systems
Hi guys, creator of lockable here - the easiest way to think of lockable is as `flock` for when you don’t have a shared file system. You can use it to control concurrent access to resources or to ensure only a single instance of a process runs at any given time.
Your processes can acquire, refresh and release locks via simple HTTP requests, so it’s language/framework agnostic. E.g. with `curl`:
$ curl https://lockable.dev/api/acquire/my-lock-name
{
"response": true //or false, if the lock wasn’t available
}
$ curl https://lockable.dev/api/release/my-lock-name
There’s also a Python client[0], which makes using the service a more pleasant experience.
Feel free to play around, the free tier is fully functional. Happy to hear any feedback you might have.
[0]: https://docs.lockable.dev/en/latest/python-client/ https://docs.lockable.dev/en/latest/python-client/
- deleted 4y ago[deleted]
- maest 4y agoWhat's the difference between this and e.g. using a Postgres db?
- keyless_ 4y agoProbably ease of setting up and not needing to manage a piece of infrastructure - which is the case with all service offerings, really. It's not a difficult conceptual task to keep track of some locks in a Postgres DB (or use PG advisory locks), but you still need to: * make sure all processes can access the db (directly or indirectly) * make sure the db can handle all the connections (or set up PGBouncer if you think you're going to be handling many processes at the same time) * write some client-side logic to acquire locks, retry on failure etc.
- skyde 4y agothey are both equally unsafe to use as a storage for distributed lock. unless the only thing you are trying to protect with your lock is access to other rows in the same postgresql server
- acisma 4y ago
- wizwit999 4y agoI recall using https://github.com/awslabs/amazon-dynamodb-lock-client https://github.com/awslabs/amazon-dynamodb-lock-client previously, this is basically a managed version. Interesting.
- keyless_ 4y agoIndeed, the first version of this heavily borrowed from this approach. The trickiest part is correctly writing the logic that spawns the secondary thread used for heartbeats.
- skyde 4y agoso it’s just a lease server but without the strong linearizability guarantee you would get with ETCD and zookeepers?
- skyde 4y agobasically if it does not use synchronous replication. and master server switch then 2 client could think they own an unexpired lock. But if it does synchronous replication without using a consensus algorithm like Paxos or Raft then the system become unwrittable if an instance go down.
- keyless_ 4y agoLockable itself isn't distributed (or, at the very least, doesn't appear distributed from the outside), so I'm not sure if linearizability applies here, but maybe I'm misunderstanding you comment. In a similar vein, I guess you can ask what happens if a client first checks if a lock is available then tries to acquire it, in two separate steps; but in that case, there's no guarantee that the lock wasn't acquired in between checking and acquisition.
- tlhunter 4y agoThis is basically the SETNX Redis command, as a service. Redis servers are already quite the cheap commodity. I think the success of this service really all comes down to performance, uptime, and scalability. There's a lot of overhead with HTTP(S) which will certainly hurt perf. https://redis.io/commands/setnx/ https://redis.io/commands/setnx/
- pas 4y agodoes http/3 and/or gRPC help with that?
- KaiserPro 4y agoWhat kind of performance can one expect?
- samsquire 4y agoHow do you ensure that a lock isn't acquired by two requests? Do you use atomic compare and set? How do you release a lock reliably? How do you solve the problem of releasing accidentally while using a resource? Can the lock jam locked if the process dies? I would use Consul for this or I would try avoid needing to lock to begin with. Even better is to use a language such as bloom Lang.
- keyless_ 4y agoGood questions: > How do you ensure that a lock isn't acquired by two requests? Indeed, the compare and set is done atomically. It's guaranteed that the lock can only be acquired by at most one process. > How do you release a lock reliably? How do you solve the problem of releasing accidentally while using a resource? A lock is released in one of two conditions: 1. the /release endpoint is called 2. the lease on the lock expires I'm not sure what you mean by "releasing accidentally" - if nobody calls /release then the lock won't be released. > Can the lock jam locked if the process dies? Locks come with a lease which expires after a set amount of time. If lockable doesn't receive a heartbeat to renew the lease, the lock is released automatically. > I would use Consul for this or I would try avoid needing to lock to begin with. For sure Consul is an alternative, so are ZooKeeper and things like ETCD - lockable is intended to be a no-setup alternative to something like that.
- samsquire 4y agoI'm thinking of a case where a process thinks it has a lock and tries to do things even though it doesn't have the lock. This happened on a project I was on, two processes would join a RabbitMQ on a service that was not safe to load balance. I suspect it could happen if the program using your lock service is not implemented properly and the lease expires but the program doesn't realise.
- keyless_ 4y agoUltimately, you have to trust your clients to do the right thing. The lockable server guarantees that e.g. if multiple /acquire requests come in for the same lock in a short time span, only one request will be successful. You are correct in that, without care, there can be pathological cases where a client may not realize their lease has expired.
- JoshTriplett 4y agoHave you considered a version that holds a lock as long as it maintains an HTTP connection (with periodic data sent to keep the connection alive), and when the connection closes, the lock is released? That would help prevent the lock from staying held.
- keyless_ 4y agoConceptually, that's how it works, however, instead of keeping an HTTP connection open, lockable expects to receive periodic heartbeats /hearbeat to keep the lock acquired. The TTL is variable, and can be set depending on the use case.
- diggs 4y agoThis is a misleading and dangerous service. You provide a distributed lease, not a lock. A distributed lease by itself doesn’t provide mutual exclusion. Distributed leases are typically accompanied with a fencing token (which your service cannot provide out of band) or an optimistic lock on the underlying "exclusive" resources (which could be implemented by the consumer of your service). I think of distributed leases as an optimization e.g. they provide soft exclusion in the happy path, which may reduce thrashing on the underlying real mutual exclusion mechanism (like an optimistic lock) under general use. Your docs and terminology guide users into building incorrect systems e.g “ensure only a single instance of a process runs at any given time” is simply not true. edit: I had a quick look at your Python library, and while Python isn't my forte, it looks like it can be trivially induced to break the mutual exclusion mirage: the "requests" lib's default request timeout is None e.g infinite, so what happens when `client.try_heartbeat` goes out to lunch indefinitely? Looks like `locks._run_heartbeat_loop` will hang, the lease will never be renewed, and within 60 seconds a competing lease holder will claim it. Boom. So you fix the timeout issue and problem solved right? Still no dice... what happens when you hit a pathological hang in your non-real-time runtime or OS? Boom.
- that_guy_iain 4y ago> This is a misleading and dangerous service. How is it dangerous? What danger would the user be in?
- mmcallister 4y agoBuilding software that relies on an incorrect assumption sounds dangerous to me. There's plenty of critical software in the world, some of which has real world consequences if there are bugs.
- HWR_14 4y agoThe user could create software that thinks a resource is locked under their exclusive control, but is not. This can lead to data corruption, which could lead to even errors propagating throughout a system.
- 4y ago
- vore 4y agoAs these locks automatically expire, it seems scary to me that losing network connectivity for >TTL can break mutual exclusion guarantees.
- keyless_ 4y agoUnfortunately, you do need _some_ sort of TTL because there is no way for the lockable server to know when a remote process has simply died. However, TTLs can be set to be arbitrarily long (e.g. multiple days), which means, in practice, you can avoid losing locks due to networking issues.
- IanCal 4y agoYou can instead have manual overrides. Not seeing a lock means that at least a person has decided it's safe to run Vs something simply have taken too long to respond.
- GordonS 4y agoIf TTLs can be arbitrary lengths, wouldn't setting them to high values (a week, month, year, whatever) allow you to implement whatever manual override mechanism you wanted?
- henrydark 4y agoUnlike locks, can a second instance call the heartbeat api without acquiring the lock?
- keyless_ 4y agoYes, any process can call a heartbeat. If the lock hadn't been acquired (by any process), the heartbeat call fails.
- GordonS 4y agoRandom thought - doesn't S3 have a leasing/locking system that could be used for distributed locks?
- keyless_ 4y agoUnfortunately, it doesn't. That was actually the reason I built this originally - there's no easy way to control read/write access to a file on S3 without using some sort of external locking system.
- killingtime74 4y agoDid Zookeeper or Etcd not suit your needs?
- keyless_ 4y agoThey are valid solutions for this use-case; the main drawback is you need to set up and maintain a ZooKeeper cluster. Lockable is intended as a simpler and faster-to-use alternative.
- IanCal 4y agoYou can add locks to objects, which restricts who can delete them and how, but requires bucket wide settings.
- 4y ago
- thatha7777 4y agoThis is misleading and a bit dangerous... You'd also need to "sync" the lock resource, which is accessed over REST. At the very least, you need some sort of Idempotency-Key for your REST API. (Imagine the scenario where /acquire succeeds on the server, but the network response fails before the client gets to read it. The client retries. How does the server know it's the same request?)
- klysm 4y agoIf you’re using distributed locks/leases you probably messed up somewhere. I think there’s no way you write correct code using this.
- sriram_malhar 4y agoThis can be a useful service, but it needs durability to avoid race conditions. If the service goes down and restarts, it will it lose all locks currently held. At this point, a new client could obtain the lock, while an older client that is still blissfuly unaware of the service going down may proceed assuming that its lease is still valid. If it is implemented as a single server, then it must persist the lock info to disk before granting a lock. Or one can get durability using replication. In which case I would just as soon use zookeeper.