5 ms·
Hey HN, I built this. Honker adds cross-process NOTIFY/LISTEN to SQLite. You get push-style event delivery with single-digit millisecond latency without a damon
by russellthehippo 6mo ago
Hey HN, I built this. Honker adds cross-process NOTIFY/LISTEN to SQLite. You get push-style event delivery with single-digit millisecond latency without a damon/broker, using your existing SQLite file. A lot of pretty high-traffic applications are just Framework+SQLite+Litestream on a VPS now, so I wanted to bring a sixer to the "just use SQLite" party.
SQLite doesn't run a server like Postgres, so the trick is moving the polling source from interval queries on a SQLite connection to a lightweight stat(2) on the WAL file. Many small queries are efficient in SQLite (https://www.sqlite.org/np1queryprob.html https://www.sqlite.org/np1queryprob.html) so this isn't really a huge upgrade, but the cross-language result is pretty interesting to me - this is language agnostic as all you do is listen to the WAL file and call SQLite functions.
On top of the store/notify primitives, honker ships ephemeral pub/sub (like pg_notify), durable work queues with retries and dead-letter (like pg-boss/Oban), and event streams with per-consumer offsets. All three are rows in your app's existing .db file and can commit atomically with your business write. This is cool because a rollback drops both.
This used to be called litenotify/joblite but I bought honker.dev as a joke for my gf and I realized that every mq/task/worker have silly names: Oban, pg-boss, Huey, RabbitMQ, Celery, Sidekiq, etc. Thus a silly goose got its name.
Honker waddles the same path as these giants and honks into the same void.
Hopefully it's either useful to you or is amusing. Standard alpha software warnings apply.
- hk1337 6mo agoI love the name!
- russellthehippo 6mo agohonk
- arowthway 6mo agoNice, I had no idea that stat() every 1 ms is so affordable. Aparently it takes less than 1 μs per call on my hardware, so that's less than 0.1% cpu time for polling.
- WJW 6mo ago"Syscalls are slow" is only mostly true. They are slower than not having to cross the userspace <-> OS barrier at all, but they're not "slow" like cross-ocean network calls can be. For example, non-VDSO syscalls in linux are about 250 nanoseconds (see for example https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/ https://arkanis.de/weblog/2017-01-05-measurements-of-system-...), VDSO syscalls are roughly 10x faster. Slower than userspace function calls for sure, but more than affordable outside the hottest of loops.
- slashdev 6mo agoThat’s ignoring the other costs of syscalls like evicting your stuff from the CPU caches. But I agree with the conclusion, system calls are still pretty fast compared to a lot of other things.
- vlovich123 6mo agoSmall correction on ambiguous wording - syscalls do not evict all your stuff from CPU caches. It just has to page in whatever is needed for kernel code/data accessed by the call, but that’s no different from if it was done in process as a normal function call.
- Polizeiposaune 6mo agoDepending on implementation details of your CPU and OS, the syscall path may need to flush various auxillary caches (like one or more TLBs) to prevent speculation attacks, which may put additional "drag" on your program after syscall return.
- vlovich123 6mo agoCorrect but you’d also still have that drag just from the kernel dirtying those caches in the first place. But I was clarifying because the wording could be taken as data/instruction cache and there generally isn’t a full flush of that just to enter/leave kernel.
- andersmurphy 6mo agoIs the main use case for this for languages that only have access to process based concurrency? Struggling to see why you would otherwise need this in java/go/clojure/C# your sqlite has a single writer, so you can notify all threads that care about inserts/updates/changes as your application manages the single writer (with a language level concurrent queue) so you know when it's writing and what it has just written. So it always felt simpler/cleaner to get notification semantics that way. Still fun to see people abuse WAL in creative ways. Cool to see a notify mechanism that works for languages that only have process based concurrency python/JS/TS/ruby. Nice work!
- infogulch 6mo agoHe mentions Litestream, maybe this also works for litestream read-only replicas which may be in completely different locations?
- russellthehippo 6mo agoWhoa I really hadn’t considered this. Do a litestream read replica, trigger across machines with S3 as the broker essentially. But you’re still stuck with the litestream sync interval. Maybe interesting for cross server notify?
- infogulch 6mo agoI guess the idea is to have all writes go through a central server with local read replicas for improved read perf. The default litestream sync interval is 1s. I bet many use-cases would be satisfied with a few seconds delay for cross-region notifications.
- russellthehippo 6mo agoIt's good for pubsub but not for claim/ack workflow unless you do If-None-Match CAS semantics on a separate filesystem which, actually, yeah that's probably fine. Feels heavy on S3 ops. But! you do save on inter-AZ networking, the Warpstream hypothesis.
- infogulch 6mo agoNeat idea! Would it help if subscriber states were also stored? (read position, queue name, filters, etc) Then instead of waking all subscription threads to do their own N=1 SELECT when stat(2) changes, the polling thread could do Events INNER JOIN Subscribers and only wake the subscribers that match.
- noveltyaccount 6mo agoThis is really interesting. I'm building something on Postgresql with LISTEN/NOTIFY and Postgraphile. I'd love to (in theory) be able to have a swappable backend and not be so tightly coupled to the database server.
- ncruces 6mo agoProbably missing something, why is `stat(2)` better than: `PRAGMA data_version`? https://sqlite.org/pragma.html#pragma_data_version https://sqlite.org/pragma.html#pragma_data_version Or for a C API that's even better, `SQLITE_FCNTL_DATA_VERSION`: https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntldataversion https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sql...
- psadri 6mo agoFor one it seems to be deprecated.
- ncruces 6mo agoIt's not.
- russellthehippo 6mo agoYep, definitely still in use. Do yall above have an opinion if the pragma is better than the syscall? What are the trade offs there? Another comment thread mentioned this as well and pointed to io uring. I was thinking that dism spam is worse than syscall spam.
- ncruces 6mo agoDepends on what to mean by better. I may be wrong, but I think you wrote somewhere that you're looking at the WAL size increasing to know if something was committed. Well, the WAL can be truncated, what then? Or even, however unlikely, it could be truncated, then a transaction comes and appends just enough to it to make it the same size. If SQLite has an API it guarantees can notify you of changes, that seems better, in the sense that you're passing responsibility along to the experts. It should also work with rollback mode, another advantage. And I don't think wakes you up if a large transaction rolls back (a transaction can hit the WAL and never commit). That said, I'm not sure what's lighter on average. For a WAL mode database, I will say that something that has knowledge of the WAL index could potentially be cheaper? That file is mmapped. The syscalls involved are file locks, if any.
- rich_sasha 6mo agoPretty cool! I have a half baked version of something similar :) Can you use it also as a lightweight Kafka - persistent message stream? With semantics like, replay all messages (historical+real time) from some timestamp for some topics? As with pub/sub, you can reproduce this with some polling etc but as you say, that's not optimal.
- russellthehippo 6mo agoAbsolutely! That’s the durable pubsub angle for sure.
- grumbelbart2 6mo agoVery cool! Another maybe stupid question, would something like inotify(7) help to get rid of any active polling?