7 ms·
Show HN: Goqite, a persistent message queue Go library built on SQLite
I wanted to build a persistent message queue based on SQLite, because that's what I'm using for my main state anyway. This gives me ACID across state and messaging, which is nice!
I've been inspired by the terminology of AWS SQS for this, but it's obviously much simpler.
Maybe you can use it too. :)
- lifeinthevoid 3y agoIn the example the user has to create the schema and manually open the SQLite connection, can't you abstract that away?
- abound 3y agoNot the author, but I imagine most projects already have a way of managing their SQLite schema (e.g. a migration system like go-migrate or goose) that they'd integrate the new table(s) into.
- markusw 3y agoThis exactly. If you're using SQLite in your Go project, you most probably already have your sql.DB and schema migrations sorted, so I don't want to get in the way of that. :) (But I can see how that could be clearer from the example code. I think I'll add that as comments. Thank you for the feedback!)
- markusw 3y agoI've updated the code example to reflect this.
- whalesalad 3y agoWouldn't you want to use a different db file for this? I wouldn't want to put this in the same db as the rest of my app, especially considering how simple it is to just open a new file.
- markusw 3y agoIf you want to be able to send to the queue in the same transaction as other state changes, it has to be the same file. That’s often a useful property.
- lifeinthevoid 3y agoSQLite allows transactions involving multiple databases (see https://www.sqlite.org/atomiccommit.html https://www.sqlite.org/atomiccommit.html section 5)
- markusw 3y agoOh! TIL! Thank you for that. :)
- yard2010 3y agoSQLite is such a great thing. Amazing value for zero cost both shmeckels and maintenance. It's a shame most startups go for postgres blindly due to "scalability concerns" it feels like a major case of premature optimization.
- markusw 3y agoIt really is! I've become a SQLite fanboy. :D Although I can definitely see the value in Postgres as well. But it's usually one of those two.
- pdimitar 3y agoMy beef with SQLite is the weaker typing + not even enforcing foreign keys unless you open the DB in a certain mode. IMO it should become more opinionated and stricter. I'd honestly pay to buy a license for SQLite with the legacy baggage removed and rewritten to be as strict as PostgreSQL or more. Well... basically embedded PostgreSQL I guess.
- markusw 3y ago
- emadda 3y agoHow is this better than just using SQLite transactions directly and a task queue table?
- markusw 3y agoIt has a Go API. ;) Check the source, that's essentially what it is. But you get this particular design, the tests etc. for free, so if you value your time, maybe this is worth something to you. Maybe not though. Either way, it's fine.
- rco8786 3y agoLooks neat, but how on earth is "Goqite" meant to be pronounced?
- markusw 3y agoThanks! See the first sentence of the readme. ;) "goqite (pronounced Go-queue-ite) is…"
- ncruces 3y agoYou should upcase the Q then. ;)
- markusw 3y agoHehe. Perhaps.
- jareklupinski 3y agomaybe GoQuette? (like eti-quette or cro-quette, go-quette)
- markusw 3y agoNah, I've already got the domain. :D
- foxbee 3y agoAnnoying but I felt exactly the same. I'd advise you to rename the project - it's still early days and the longer it goes on the harder it will be to change - I've felt this pain.
- packetlost 3y agoI made something extremely similar, but lighter weight awhile ago: https://github.com/chiefnoah/goalpost https://github.com/chiefnoah/goalpost It uses bbolt/bolt instead of sqlite. It was for use on shared servers that had questionable uptime. I haven't touched it in a long time, but it served its purpose well
- markusw 3y agoYeah, there are a lot of queue libraries like ours. :) It was important to me it was SQLite, since that's what I use as the storage layer already.
- packetlost 3y agoYeah, not trying to detract from your work! At the time when I made mine (like, 6 years ago) there was nothing in the space. Now there's plenty! Nice work!
- markusw 3y agoDidn’t take it that way, but thank you for the clarification. :) And thanks! Likewise. :D
- slicedbrandy 3y agoAlso did something very similar, but swapped out the storage layer with an embed of leveldb. Also supports both an HTTP and Redis API. https://github.com/tomarrell/miniqueue https://github.com/tomarrell/miniqueue
- markusw 3y agoInteresting that it supports the Redis API as well! As noted elsewhere here, it was important to me to use SQLite, since that's what I use as the storage layer already. But I've added HTTP adapters for remote clients. I hadn't thought of using a different remote API, interesting idea. Thank you for the pointer!
- pstuart 3y agoI love Go and SQLite and enjoy using them together. One bummer is that it requires CGO to use and that makes cross compiling a pain. It might be nice to use the modernc variant (via build tags) to make this more portable -- https://gitlab.com/cznic/sqlite https://gitlab.com/cznic/sqlite
- markusw 3y agoIt’s bring-your-own-driver, so you can use a CGo-free one. :) The one in go.mod is only used in tests.
- willi59549879 3y agoZig allows cross compilation of cgo
- pdimitar 3y agoThere is a pure Golang SQLite implementation btw, I love that aspect of the Golang ecosystem. Not even Rust has that and I am envious.
- pstuart 3y agoGive Rust some time, there's a wee bit of competition going on there ;-)
- pdimitar 3y agoOh, absolutely, it's just kind of strange for me that nobody has bothered translating the SQLite C code to Rust, and Golang's community did. Though I'd guess tools like `c2rust` don't produce idiomatic Rust code and it can be a lot of effort to turn that into proper Rust. That's likely the biggest reason.
- heipei 3y agoCool idea, I can definitely see the appeal of having everything in one database. I would probably also add an "attempts" column (int) so you can discard messages after X attempts.
- markusw 3y agoThat’s the “received” column. :) Messages after X attempts aren’t received by the normal receiver method anymore.
- gigatexal 3y agoAs much as I love SQL I would have thought this would have abstracted away the SQL bit of the queue and just given me high level queue methods to do stuff with. I don’t like this much for that reason.
- markusw 3y agoI don't understand your comment. Abstracting away the SQL is exactly what this library does (and pretty much the only thing): Send, Receive, Delete methods. Could you clarify?
- echoangle 3y agoWell your example on the website contains a Database schema and manually opens a DB connection. Why do I as a library user need to care about that? Just do that internally when creating the queue instance
- markusw 3y agoSee this other thread: https://news.ycombinator.com/item?id=39670199 https://news.ycombinator.com/item?id=39670199
- janderland 3y agoFrom the example, it appears you still need to setup schema yourself using SQL. This makes the library a leaky abstraction because it only partially hides the SQL.
- asa400 3y agoThis is super cool! I like your method for handling the claiming of queue items. I always end up wishing SQLite had something like Postgres' LISTEN/NOTIFY/SKIP LOCKED but your UPDATE solution is nice and low tech and fitting for SQLite, in a good way!
- markusw 3y agoThank you, I’m really glad you like it! :)
- polyrand 3y agoI also built something similar for Python [0]. I decided to go with a separate DB file for the queue, I think it's better for performance but also to avoid mixing the data storage with the message queue (in my case, I assume the message queue is just an implementation detail and not really tied to the rest of the DB schema). [0]: https://github.com/litements/litequeue https://github.com/litements/litequeue
- markusw 3y agoWell peepz, it’s been fun, but I’m off to bed now because sometimes I get sleepy. Ta! :D (Thank you for all the comments!)
- dev-tacular 3y ago> says they are sleepy > 30 minutes later they open a new PR to address a comment left by [an HN user](https://news.ycombinator.com/item?id=39673795 https://news.ycombinator.com/item?id=39673795) I think this person might like coding
- markusw 3y agoHaha, you got me. :D
- Ingon 3y agoIn a me too way, I also created something similar - https://github.com/klev-dev/klevdb https://github.com/klev-dev/klevdb It started on top of bbolt, but over time moved a completely custom implementation. The main reason for this was performance, but I also wanted to run a lot of queues, so each one had to be as light as possible. On top it even provides some indexing capabilities, which (when enabled) allow for quick find by key and time.
- Xeoncross 3y agoInteresting, the performance of https://github.com/klev-dev/klevdb https://github.com/klev-dev/klevdb is 10x https://github.com/maragudk/goqite https://github.com/maragudk/goqite so it makes me assume the durability is somewhat lacking. Can you speak to the tradeoffs here around message loss? I would think that having a small chance of message loss due to writing to an append only log in batches might be a reasonable trade off for many things (if that is how it works).
- Ingon 3y agoFirst off, looking at goqlite benchmarks, we are measuring different things - goqlite is measuring a complete lifecycle of a message, which includes sending, receiving and deleting. The klevdb benchmarks are usually exercising one (at most 2) of these. In klevdb you can control the fsync yourself - using AutoSync you can ask it to perform fsycn after each Publish, which currently is best effort put on disk. Of course in this case performance suffer dramatically (as you would expected, esp if you don't batch). You can also not use AutoSync and control it yourself (klevdb exposes Sync), deciding what kind of message loss you are prepared to deal with. The benchmarks you see run without AutoSync, to highlight what kind of raw throughput klevdb is capable of. And yeah, klevdb is basically an append-only log. Of course, you can delete messages from it, but this is done in parallel, where the segment is being copied without the messages you want to delete, then replaced with a single mv system call. Rewriting the head segment (e.g. the one you are writing to) is not always possible (since there could be messages the rewriter haven't seen), but will eventually succeed (once the head segment is no longer a head). Hope this answers your questions, and if not, happy to expand more. I've been contemplating on implementing transactions on top of klevdb, but haven't quite figured out if I need them or not. Edit: spelling and grammar
- mattrighetti 3y agoInteresting, this seems to be something that a lot of people tried to implement, it’s the first time I see this many “I did something similar” under a project! Cool project! Would love to implement something similar in another language and maybe compare performances, even though I guess they would be very similar since there’s gonna be SQLite under the hood.
- markusw 3y agoYeah, it's not exactly a new idea. I'm actually a bit surprised it's this popular. :D Thanks! Doing it in another language would be almost trivial, there's really not a lot of code involved. I've focused mostly on making a nice, easy to use API for the common cases. No more, no less.
- cyanydeez 3y agoGoqlit might've been closer a brand-action name
- markusw 3y agoSee the other thread on naming. ;)