13 ms·

Show HN: Drop-in SQS replacement based on SQLite

656 points by memset 2y ago

Hi! I wanted to share an open source API-compatible replacement for SQS. It's written in Go, distributes as a single binary, and uses SQLite for underlying storage. I wrote this because I wanted a queue with all the bells and whistles - searching, scheduling into the future, observability, and rate limiting - all the things that many modern task queue systems have. But I didn't want to rewrite my app, which was already using SQS. And I was frustrated that many of the best solutions out there (BullMQ, Oban, Sidekiq) were language-specific. So I made an SQS-compatible replacement. All you have to do is replace the endpoint using AWS' native library in your language of choice. For example, the queue works with Celery - you just change the connection string. From there, you can see all of your messages and their status, which is hard today in the SQS console (and flower doesn't support SQS.) It is written to be pluggable. The queue implementation uses SQLite, but I've been experimenting with RocksDB as a backend and you could even write one that uses Postgres. Similarly, you could implement multiple protocols (AMQP, PubSub, etc) on top of the underlying queue. I started with SQS because it is simple and I use it a lot. It is written to be as easy to deploy as possible - a single go binary. I'm working on adding distributed and autoscale functionality as the next layer. Today I have search, observability (via prometheus), unlimited message sizes, and the ability to schedule messages arbitrarily in the future. In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues. I'd love your feedback!

localfirst 2y ago
[flagged]
- pvg 2y ago
  The purpose of Show HN is not really ‘product catalogue’
  - localfirst 2y ago
    i didnt ask for a product catalogue, all im asking is to clarify whether you are truly FOSS or not by being transparent about the license. Any type of licensing conditions like AGPL 3.0 to force you to share your code is a no go for many many companies. I understand the desire to make money and you can do that without AGPL 3.0
    - bumbledraven 2y ago
      It's right there in the LICENSE file: https://github.com/poundifdef/SmoothMQ/blob/main/LICENSE https://github.com/poundifdef/SmoothMQ/blob/main/LICENSE What's unclear or not transparent about that?
      - localfirst 2y ago
        All I'm asking is to include the type of license in the Show HN title I had no idea there would be so much push back against this.
        ncruces 2y ago
        I actually agree. The AGPL, for something like this, is a total non-starter, IMO, and just changes the conversation. It'd be one thing to choose the AGPL because you believe in the FOSS movement. Or if you're releasing something end-users can benefit from, directly, by self hosting it. To release something that will always be a component of something bigger, license it as AGPL, then talk about monetization… just cut the chase and release it as “source available,” with a licence people can actually use, even if it has a bunch of not-really-FOSS strings attached. Because who can realistically use this as is? Who can download the source and actually do something with it? What AGPL-compatible FOSS project is dying to use a drop-in AWS SQS replacement?
        localfirst 2y ago
        Just from IP/legal point of view AGPL and AGPL adjacent licensing that imposes control over how you use the code is a non-starter for many companies especially ones that are "backed by YC" proudly displayed on their website because that signals money will be exchanged in the future by design. MIT/BSD license that imposes no control over the how the code is used is the only FOSS we are allowed to use and we do donate regularly to maintainers working selflessly. What makes me angry is that somebody then uses that MIT/BSD license code, makes some modifications and releases it as AGPL with clear intent to make a buck from it without paying that original maintainer any money.
- deleted 2y ago
  [deleted]
philippta 2y ago
One quick suggestion on project structure: Move all the structs from models/ into the root directory. This allow users of this package to have nice and short names like: q.Message and q.Queue, and avoids import naming conflicts if the user has its own „models“ package.
- memset 2y ago
  Thanks for the tip - and for even looking at the code! I always struggle to figure out how to organize things.
  - philippta 2y ago
    Just noticed that your root directory is already „package main“ so you can either move that to /cmd/something/ or simply rename models/ to q/. That would have the same effect and is also idiomatic.
- pqb 2y ago
  Actually, the "model/" (note: singular form) directory (package model) would be preferred in the golang world.
mannyv 2y ago
So I assume it does the back-end as well? I never cared to figure out what parts of SQS are clients-side and server side, but - does SmoothMQ support long polling, batch delivery, visibility timeouts, error handling, and - triggers? Or are triggers left to whatever is implementing the queue? Both FiFo and simple queues? Do you have throughput numbers? As an SQS user, a table of SQS features vs SmoothMQ would be handy. If it's just an API-compatible front-end then that would be good to know. But if it does more that would also be good to know. The reason you'd use this is because there are lots of clients who still want on-prem solutions (go figure). Being able to switch targets this way would be handy.
- memset 2y ago
  Everything here is on the backend. The client does very little except make api calls. It implements many of these features so far (ie, visibility timeouts) and there are some that are still in progress (long polling.) a compatibility table is a good idea.
deskr 2y ago
Perhaps do a small example application. go get github.com/poundifdef/SmoothMQ/models go: github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60 requires go >= 1.22.2; switching to go1.22.4 go: github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60 (matching github.com/poundifdef/SmoothMQ/models@upgrade) requires github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60: parsing go.mod: module declares its path as: q but was required as: github.com/poundifdef/SmoothMQ
- memset 2y ago
  Good idea! fyi this is not meant to be used as a library. It runs as a standalone server, and then your application connects to it using an existing AWS sdk.
xchip 2y ago
Thanks! Loving all these self-hosted KISS stuff :)
encoderer 2y ago
I think it’s better to say it’s sqs compatible versus an sqs replacement.
pqdbr 2y ago
How would one use this to replace Sidekiq for instance? Rate limiting is something I miss on Sidekiq (only available on the premium plan that I can't afford) and the gems that extend it break compatibility often.
koito17 2y ago
I may be asking a naive question, but what is the rationale behind disabling foreign key support and using them anyway in the database schema? See https://github.com/poundifdef/SmoothMQ/blob/46f8b22/queue/sqlite/sqlite.go#L56-L88 https://github.com/poundifdef/SmoothMQ/blob/46f8b22/queue/sq... The "TODO: check for errors" comment, combined with what seems like disabling foreign key constraint checks, makes me a bit hesitant to try this out.
- memset 2y ago
  Good question. It was an evolution. Originally I enforced foreign keys, but inserts/updates were unbearably slow. So I updated the connection string to disable them (but, as you point out, I haven't updated the CREATE TABLE statements.) In practice, I did not find they were necessary - I was only using foreign keys to automatically CASCADE deletes when messages were removed. But instead of relying on sqlite to do that, I do it myself and wrap the delete statements in transactions. There are many TODOs and error checkings that I will, over time, clean up and remove. I'm glad you've pointed them out - that's the great thing about open source, you at least know what you're getting into and can help shine a light on things to improve!
  - plasma 2y ago
    Some of the slowdown will come from not indexing the FK columns themselves, as they need to be searched during updates / deletes to check the constraints.
john-tells-all 2y ago
Why not use LocalStack? It has SQS and a lot of AWS services for testing/development. Well documented, open source. https://docs.localstack.cloud/overview/ https://docs.localstack.cloud/overview/
- crazysim 2y ago
  I think the vibe might be that this is something someone could use for production.
  - mtndew4brkfst 2y ago
    With no distribution model and built on SQLite, it certainly doesn't scream "production candidate" to me.
    - ku1ik 2y ago
      There’s more production systems on this planet running on a single node than ones running in a distributed setup. And SQLite is a rock solid database, deployed in more places than Postgres. The only thing here that could be taken when weighing production readiness would be the maturity and stability of this project itself.
    - gkbrk 2y ago
      You know there are millions of production deployments of SQLite right? It's perhaps the most widely deployed database.
  - mannyv 2y ago
    "Production" is a big word to use for a piece of software. There are different levels of "production ready," depending on what you care about. Some enterprise production questions: * Is it HA? What HA topology is it designed for? * How many messages-per-second can it ingest? * What about with multiple clients? * Can you front-end it via a load balancer? * Can it guarantee only-one delivery? * How does it handle errors? ie: when a message is retrieved by a client and the client errors after the server dies, what happens to the message - does it redrive or expire correctly when the server comes back up? * How does it guarantee consistency in a multi-node environment? * How dies it behave when a node dies/runs out of memory? * How do you monitor it? * How do you recover a corrupted instance? * what happens if/when the SQLite partition runs out of space? etc etc. I use SQS as a main pipeline, and I would want to know a bunch of these things before I'd even consider replacing it. But for someone with one box on a VPS somewhere, you probably only care if it runs for a year without crashing
neuroscisoft 2y ago
This is very interesting! The self-hosted aspect is something I'll have to consider for certain purposes. My lab also developed an SQS-esque system based on the filesystem, so no dependencies whatsoever and no need for any operational system other than the OS. It doesn't support all SQS commands (because we haven't needed them), but it also supports commands that SQS doesn't have (like release all messages to visible status). https://github.com/seung-lab/python-task-queue https://github.com/seung-lab/python-task-queue
threecheese 2y ago
There is a lot of new ecosystem being built around SQLite in the browser using wasm, as a primary data store or as a local replica for every client, and there are some interesting interactions with crdt and peer to peer applications; does this fit into your business case? Would be interesting to see a massively distributed - via browser embedding - queue, that even uses standard sqs bindings.
andrewstuart 2y ago
I wrote sasquatch which is similar. sasquatch is more simple than SQS and queues are virtual (i.e the queue exists because you put a message in it, and is gone when there's no messages in it). https://github.com/crowdwave/sasquatch https://github.com/crowdwave/sasquatch sasquatch is also a message queue, also written in Golang and also based on sqlite. sasquatch implements behaviour very similar to SQS but does not attempt to be a dropin replacement. sqsquatch is not a complete project though, nor even really a prototype, just early code. Likely it does not compile. HOWEVER - sasquatch is MIT license (versus this project which is AGPL) so you are free to do with it as you choose. sasquatch is a single file of 700 lines so easy to get your head around: https://raw.githubusercontent.com/crowdwave/sasquatch/main/sasquatch.go https://raw.githubusercontent.com/crowdwave/sasquatch/main/s... Just remember as I say it's early code, won't even compile yet but functionally should be complete.
PanMan 2y ago
Correct me if I’m wrong but: SQLite sounds like: runs on one server. While that will work most of the time, it won’t work 100% of the time. I don’t know the specifics, but I’m fairly sure if a queue server crashes, SQS will keep working, as stuff is redundant. So while it can work in a best case, this (probably) won’t have the same reliability as SQS has..
- memset 2y ago
  I break it down two ways. First, the project does not yet have a distributed implementation, you’re correct. Stay tuned! Second, SQLite is incidental. Data is still stored on disk, just as SQS must be, but I’ve chosen SQLite as the file format for now.
  - 22c 2y ago
    Strictly speaking, SQS does not necessarily store data on disk and being highly available and fault tolerant does not exclude one from operating a completely volatile data store. Indeed, SQLite itself has an in-memory option which, when appropriately architected, could be used as the primary data store for a highly available and fault tolerant queue. Having said that, I think it is safe to assume that SQS probably stores information to non-volatile storage. Nice work on the project!
- parhamn 2y ago
  The choice of SQLite is a great one as the distributed sqlite space is also growing. You have things like rqlite (raft+sqlite), and you have cloudservices like CloudFlare D1 which is basically distributed/HA sqlite. Plus you can swap it for basically any other sql database.
- jpc0 2y ago
  Does this even need to compete feature wise with SQS? If it faithfully reproduces the SQS Api what could possibly stop me from using this product now (if he ever does hosted) and then switching to SQS if the scale is ever justified? I'm all for a full suite of solutions targeting the same API. Can I run this thing on a single server with my dashboard app for the 3 person team I'm developing for and all that it cost them is per hour for tech support and server hardware and deloy my multi million user app on SQS but have effectively the same code.
ahachete 2y ago
Congratulations! I also love writing AWS API-compatible services. That's why I did Dyna53 [1] ;P (I know, unrelated, but hopefully funny) [1] https://dyna53.io https://dyna53.io
- memset 2y ago
  This is really fun!
- selcuka 2y ago
  It is fun, indeed. Could be a candidate for the best technology abuse of the year.
- bensmoif 2y ago
  This is an absolutely unhinged project idea. Great work
dav43 2y ago
Tangential but I looked at the codebase - because I like to imagine I can code (I can’t) - and for a laymen, I could follow the code. Makes me think quite positively of go lang and of the dev for designing in such a way. Can understand why teams like it because of easier maintenance Quite elegant from my uneducated pov.
- memset 2y ago
  Thank you for the kind words! I'll be sure to work on making it as complicated as I can :)
- deleted 2y ago
  [deleted]
- x-complexity 2y ago
  Honestly shows the capabilities of both: - The dev for making it both fast & simple to understand - Golang for making the codebase easy to follow
- silisili 2y ago
  That's the beauty of Go, generally, and why I immediately became attracted to it. The code is supremely readable because it enforces a style guide, and doesn't allow much magic. A number of people don't like it for limiting their expression and abilities, which I understand that feeling too. But as a middle aged programmer I realized that readability trumps conciseness and cleverness in the long run.
  - bilekas 2y ago
    I can't remember who said it but I remember hearing that a good go Dev and a new go Dev should be able to produce generally similar code in order for maintainability. I do like playing with go, it's sits in a nice little playground in my head, mostly thanks to the compile times I think.
    - ayuhito 2y ago
      Two experienced Go devs tackle the same task, and their code will look almost identical. Two experienced Rust devs tackle the same task, and their solutions will be worlds apart. (I write both, but I do love Go for its simpleness)
leetrout 2y ago
Glad to find out what you've been up to! Very cool and I am excited to see where you take this.
dracarys18 2y ago
The only reason I use SQS is because how it's compatible with other AWS Services, I can have an S3 bucket whenever there's an upload I can send an SQS Message and I can have it trigger a lambda.
TheAnkurTyagi 2y ago
sounds like a great project. How do you handle performance with large message sizes?
throwaway984393 2y ago
[dead]
michaelcampbell 2y ago
This is really pedantic, totally outside any actual functionality, and may be a me thing, but a trigger for me is seeing left-justified (or worse, centered), columns of numbers (eg: your number of messages column). I'll grant it's small to infinitesimal, but you asked for feedback.
- memset 2y ago
  Hadn't even thought about that. Now I can't unsee it. Thanks for the suggestion.
lovestaco 2y ago
I was thinking of using Redis Stream instead of SQS, what would be better Redis or SmoothMQ?
tjoff 2y ago
SQS: Amazon Simple Queue Service
- cwbrandsma 2y ago
  Thank you, I was scratching my head trying to figure that out.
  - rmbyrro 2y ago
    Why some people think we need AI everywhere to "augment" and "enrich" our "experiences". People don't bother a google search, after, what, 20 years in town?
- deleted 2y ago
  [deleted]
ninjazee124 2y ago
[flagged]
- memset 2y ago
  MongoDB is web scale :) https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
  - simonw 2y ago
    Have you run any benchmarks yet? I'm guessing it happily handles thousands of queue operations a second given that it's SQLite and Go - would be interesting to see how settings like SQLite WAL mode affect its performance.
    - memset 2y ago
      I'm using WAL mode - I'd almost given up on sqlite until I remembered to enable it. I've been playing with benchmarks, yes! I haven't written them up yet because I worry about doing them the "right" way. but since you asked, I did a quick test with a single server and single client on a t2.nano. With 3 sending threads and 5 receiving threads, and mesg size of 2kb, I can send 700 msgs/s and receiver 500 msgs/s. It is way faster on my laptop, and I have a lot of tricks that I've been playing with to improve this.
      - withinboredom 2y ago
        I had to build a queue implementation not too long ago. We used t2.nano for our benchmarks as well. Your results are in-line with ours pre-optimizations (except we are using nats jetstream for queuing). I’d also recommend reducing the number of threads as that can increase performance on single-processor machines (context switching will kill you). Try to find the sweet spot (for us, it was 2x-4x the number of cpus to threads, at least, for our workload). If you are using go, it kinda sucks at single-cpu things, and if you detect that you have a single core, you can lock a go-routine to a thread and that sometimes helps. Also, try to batch sends (aka, for-loop) to attempt to saturate the send-side so by the time you come back with your next batch they’re still messages sitting in a network buffer somewhere. For example, we have a channel that wakes up our sender, then we honest-to-god wait like 60ms in the hopes there will be more than one message to pick up — and there usually is in production.
jerrygenser 2y ago
> In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues. are you monetizing this as a separate business from: https://www.ycombinator.com/companies/scratch-data https://www.ycombinator.com/companies/scratch-data
- memset 2y ago
  Truthfully, I don't know yet - I haven't even built a paid/hosted version at all. It is related to my existing business in the sense that it deals with realtime data. But I started working this as something I wish existed as opposed to having some big VC strategy and pitch deck behind it. (Also, I appreciate all of your feedback on this a month ago! It was really helpful to encourage me to keep looking into this and also figuring out the "first" things to launch with!)
  - Kwpolska 2y ago
    Not every open source project needs to be monetized. Projects that were created to "scratch one's itch" tend to fare better than those built to make money. Devs put more love and less stress into them than into things they want to build a business out of. The monetization paragraph reads really weird, as if you believe HN is a community of VC-adjacent people looking for new ways to make money (it isn't), and talking about how you plan to exploit your new project is mandatory (it isn't either).
    - throwaway984393 2y ago
      [dead]
    - fragmede 2y ago
      > HN is a community of VC-adjacent people looking for new ways to make money it kinda is. There are all sorts of people here, but HN is owned by YC, a well known VC fund. That doesn't mean that everyone here is one, but it certainly influences the community here. > and talking about how you plan to exploit your new project is mandatory it's not mandatory but it's a frequently asked question. OP might as well answer it while they have the mic.
sgarland 2y ago
This looks great! How were you planning on tackling distributed?
- memset 2y ago
  This is a great question and something I've been thinking about a lot. Big picture: Each queue node can operate (mostly) independently, and this is good. As a consumer, I don't really care where my next message comes from, so I can minimize the amount of data that needs to have a "leader". The only data that needs to be synced is the list of queues, which doesn't change often. If one server is full, it should be able to route a request to another server. When we downscale, we can use S3/Dynamo (GCS/firestore) to store items and redistribute. There's more nitty gritty here (what about FIFO queues? What about replication?) but the fact that the main actions, "enqueue" and "dequeue", don't require lots of coordination makes this easier to reason about comapred to a full RDBMS.
  - flaminHotSpeedo 2y ago
    You're hand-waving away all the complexity in the "nitty gritty." Enqueue absolutely requires coordination, if not via leader then at least amongst multiple nodes, if you want to guarantee at least once delivery If you don't guarantee that, cool, but you're not competing with sqs
  - avinassh 2y ago
    Have you considered libsql? It is a distributed SQLite database, written mostly in Rust. There is a go database/sql and GORM driver available - https://github.com/tursodatabase/libsql https://github.com/tursodatabase/libsql disclaimer: I am one of the maintainers
    - memset 2y ago
      I did not know about libsql! Thanks for sharing. How do you think I should reason about making this distributed - should I use one of the (various) sqlite replicatoin libraries? Or should this be something I roll on my own (on top of some other protocol, such as raft, or something else.)
hrisen 2y ago
Keeping questions from scale & benchmarks aside, this is a cool thing for functional/unit testing module that uses SQS, instead of dumb mocks.
- memset 2y ago
  Thank you! Yes, you should definitely use this instead of localstack :)
  - thrauat 2y ago
    Although I wouldn't recommend using LocalStack's SQS implementation for production workloads, calling it a dumb mock is a bit outdated. It emulates pretty much every SQS behvaior including long polling, delayed messages, visibility timeouts, DLQ redrive, batch send/receive, ...
okr 2y ago
A testcontainer could be useful. I use always this adobe-mock from s3 for my test. Hmm.
- memset 2y ago
  I was not aware of testcontainer until you just mentioned it! I have a very basic Dockerfile for deploying to fly.io. What/how would I get started? It looks like localstack is a supported testcontainer and they do support SQS (but I haven't tried it myself.)
  - okr 2y ago
    Hmm, i never provided a testcontainer myself, but i have used docker images in a testcontainer. That is pretty straight-forward. The specialized testcontainers just pass down or expose more config parameters, imho, because they possess more knowledge about the docker container that it is about to start. For SQS i could imagine, that it is convenient to expose uri or arn or even auth parameters, that one then can refer to when setting up tests with dynamic config parameters and to setup the sqs client. Localstack i have not used so far. I use containers often for database tests also.
fbdab103 2y ago
Minor thing - the gif browser demo should have been maximized. The video zooming into the action on what would have fit onto a single screen was distracting. Are there maintenance actions that the admin needs to perform on the database? How are those done?
- memset 2y ago
  Thank you for the feedback - agree on the gif. Also it should probably show real messages an an example instead of rand(). re: maintenance - I have tried to build this to be hands-off. The only storage this uses is SQLite, and I have the code set to automatically vacuum as space increases. It also has a /metrics endpoint which has disk size. This is going to be used for two things in the future: first, as a metric for autoscaling (scale when disk is full) and second so that a server can stop serving requests when its disk is full (to prevent catastrophic failure.)
systemz 2y ago
Very cool. I needed something like this. Looking at short video in readme - I suspect adding live stats similar to sidekiq would make UI look more dynamic and allow quick diagnostics. Docs for current features are more important though.
- memset 2y ago
  Can you tell me more about why you needed this (and what you ended up using?) Totally agree on adding more metrics and information to the UI - but how much of that should be on the dashboard vs exposed as a prometheus metric for someone to use their dashboard tool of choice? I am not very good at visual design and have chosen the simplest possible tech to build it (static rendering + fomanticUI). I sometimes wonder if the lack of react or tailwind or truly beautiful elements will hold the project back.
stuaxo 2y ago
Good, we need open implementations of all the AWS stuff. I swear they reimplement stuff we have just so there are more places to bill us.
- paulddraper 2y ago
  SQS is cheap. 10m requests for $4. You will need hundreds of millions of requests per month for it to be noticable. And can an implementation like this even help you at that point?
  - memset 2y ago
    Question: is there any feature set (in contrast to pricing) that would make it worthwhile for someone to switch?
    - threecheese 2y ago
      Operations. At my $bigCo we are always rolling our own ops tooling for queues, as it’s more than just “reprocess message”; there is always some business logic that separates different types of consumers/destinations, and so q management is never straightforward. Might be a differentiable feature.
      - memset 2y ago
        Can you tell me more about the types of queue things you need to implement? (Feel free to email me too - I’d be grateful for the feedback!)
        threecheese 2y ago
        I’ll give you a real example, and it aligns with one of your feature goals that you mentioned elsewhere in these threads: allow you to rate limit without change on the client side. I have a workload that processes transactions for P external partners (P>1000) in a market with diverse technology sets; some partners are aggregated behind MSPs and some we integrate with directly, and so those P partners represent say 500 unique destinations. This workload handles < 100 line-of-business transaction types, spread across the partners; some support them all, some support only a few, and their implementations may be standard/consistent across transactions or their may be unique implementations. This is not a strange architecture, it represents the fact that every business-system has a lifecycle, there are legacy xml cruft and shiny graphql both exposed to the world and sitting next to each other, making money, and nearly every integrator has this. These systems go down, or networks become unavailable, or they perform maintenance, or maybe they can only support 2 concurrent connections - and every partner is different. Some MSPs come in and get some % of the market for handling this for businesses that don’t have the capability, but there are many that do. If you were in the business of exposing products that use these services, would you want your engineers having to care about any of it? Me neither, and so it is CRITICAL that we operationalize it, which for a store-and-forward integration workload basically ends up being an operational tool for queues and their destinations. I do not however find any tools that meet our requirements, which are simply “allow an operations/support team to manage transaction T for partner P who may be hiding behind MSP M” which includes rate limiting, circuit breaker, observability, routing, deployment, billing etc etc. This is IMO tied very closely to the queue - it is in fact the workload’s primary data store - and I do not understand why queue providers don’t differentiate their offerings with this kind of management capability. If I think about a business journey from startup to enterprise, maybe a small shop does not need this - they only support a small number of transactions and a small number of partners. As they grow, this requires engineering, and so they just build the tools they need to scale. But it doesn’t have to be that way.
deleted 2y ago
[deleted]
yimpolo 2y ago
This is super cool! I love projects that aim to create simple self-hostable alternatives to popular services. I assume this would work without much issue with Litestream, though I'm curious if you've already tried it. This would make a great ephemeral queue system without having to worry about coordinating backend storage.
- memset 2y ago
  I haven't tried this with litestream! It will be worth exploring that as a replication strategy. The nice thing about queues is that backend storage doesn't really need to be coordinated. Like, you could have two servers, with two sets of messages, and the client can just pull from them round robin. They (mostly) don't need to coordinate at all for this to work. However, this is different for replication where we have multiple nodes as backups.
  - xrd 2y ago
    Since this is a single go binary I bet it would work perfectly with litestream. I use litestream in the exact same way with Pocketbase and it works perfectly (Pocketbase is also a single go binary). I'm going to give it a try and report back.
12_throw_away 2y ago
Actually pretty excited to try this, there are so many cases where I need a bare-bones (aka "simple"), local, persistent queue, but all the usual suspects (amqp, apache whatever, cloud nonsense, etc.) are way too heavy. I'll probably try poking at it directly through the HTTP API rather than an SDK ... does it need AWS V4 auth signatures or anything?
- memset 2y ago
  The only HTTP API that is exposed is actually the SQS one. (I'm not opposed to a "regular" HTTP API but the goal was to make it easy for people to use existing libraries.) If you do use your language's AWS SDK, the code handles [1] all of the V4 auth stuff. https://github.com/poundifdef/SmoothMQ/blob/main/protocols/sqs/sigv4.go https://github.com/poundifdef/SmoothMQ/blob/main/protocols/s... I'd love your feedback! Particularly the difficulties you find in running it, of which I'm sure there are many, so I can fix them or update docs.
- stackskipton 2y ago
  I'd also give a shout out to beanstalkd. I use it to teach message systems and it's great.
4RealFreedom 2y ago
The goals are a little different but I think it's worth pointing out ElasticMQ. I use it simulate sqs in a docker environment. https://github.com/softwaremill/elasticmq https://github.com/softwaremill/elasticmq
- cies 2y ago
  We use ElasticMQ to have an SQS compatible service for local development. We use it with docker-compose locally. In our remote envs we use SQS. So far I had not problems with ElasticMQ. I'm much intrigued by the small LOC count of SmoothMQ. When I compare it to ElasticMQ it's much smaller (probably by using sqlite's features). https://github.com/softwaremill/elasticmq https://github.com/softwaremill/elasticmq
- memset 2y ago
  I have looked at elasticmq but not played with it myself. You might also be interested in their benchmarks of all of the existing queues out there: https://softwaremill.com/mqperf/ https://softwaremill.com/mqperf/
  - julius 2y ago
    I also just used ElasticMQ in a docker environment. Did you see any specific downsides, when you looked at it? In which scenarios should I consider replacing it with your solution?
chromanoid 2y ago
Could you elaborate why you didn't choose e.g. RabbitMQ? I mean you talk about AMQP and such, it seems to me, that a client-side abstraction would be much more efficient in providing an exit strategy for SQS than creating an SQS "compatible" broker. For example in the Java ecosystem there is https://smallrye.io/smallrye-reactive-messaging/latest/ https://smallrye.io/smallrye-reactive-messaging/latest/ that serves a similar purpose.
- westurner 2y ago
  Where AWS is the likely migration path for an app if it needs to be scaled beyond dev containers, already having tested with an SQS workalike prevents rework. The celery Backends and Brokers docs compare SQS and RabbitMQ AMQP: https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/index.html#broker-overview https://docs.celeryq.dev/en/stable/getting-started/backends-... Celery's flower utility doesn't work with SQS or GCP's {Cloud Tasks, Cloud Pub/Sub, Firebase Cloud Messaging FWIU} but does work with AMQP, which is a reliable messaging protocol. RabbitMQ is backed by mnesia, an Erlang/OTP library for distributed Durable data storage. Mnesia: https://en.wikipedia.org/wiki/Mnesia https://en.wikipedia.org/wiki/Mnesia SQLite is written in C and has lots of tests because aerospace IIUC. There are many extensions of SQLite; rqlite, cr-sqlite, postlite, electricsql, sqledge, and also WASM: sqlite-wasm, sqlite-wasm-http celery/kombu > Transport brokers support / comparison table: https://github.com/celery/kombu?tab=readme-ov-file#transport-comparison https://github.com/celery/kombu?tab=readme-ov-file#transport... Kombu has supported Apache Kafka since 2022, but celery doesn't yet support Kafka: https://github.com/celery/celery/issues/7674#issuecomment-1204053595 https://github.com/celery/celery/issues/7674#issuecomment-12...
  - chromanoid 2y ago
    I don't get where you are pointing at. RabbitMQ and other MOMs like Kafka are very versatile. What is the use case for not using SQS right now, but maybe later? And if there is a use case (e.g. production-grade on-premise deployment), why not a client-side facade for a production-grade MOM (e.g. in Celery instead of sqs: amqp:)? Most MOMs should be more feature-rich than SQS. At-least-once delivery without pub/sub is usually the baseline and easy to configure. I mean if this project reaches its goal to provide an SQS compatible replacement, that is nice, but I wonder if such a maturity comes with the complexity this project originally wants to avoid.
rmbyrro 2y ago
I'd have named it MQLite. Congrats on finishing and delivering a project, this is quite a challenge in itself. I think SQLite can be a great alternative for many kinds of projects.
slotrans 2y ago
Not sure about the goal of providing a hosted service cheaper than SQS. SQS is already one of the cheapest services on Earth. It's pretty hard to spend more than a few bucks a month, even if you really try!
- deleted 2y ago
  [deleted]
- PanMan 2y ago
  This depends on your definition of few.. but SQS costs us more than the servers handling the messages… It’s cheap until you do billions of messages…
- memset 2y ago
  That is fair. Two things to validate from a biz perspective: 1. Is there some threshold where this would make sense financially (n billions of messages.) 2. Are the extra developer features (ie, larger message sizes, observability, DAGs) worth it for people to switch? Would love your thoughts - what, if anything, would make you even entertain moving to a different queue system?
  - GordonS 2y ago
    Not the GP, but I also think you'll struggle to compete in the hosted service space. Maybe you could add a web admin GUI as a paid add on?
  - threecheese 2y ago
    What is your target market? Cloud-native is, like the commenter said, going to be difficult to differentiate based on cost. I can see this being useful for hybrid cloud (onprem instances), functional or local testing, “localfirst” workloads, enthusiasts etc.
    - memset 2y ago
      Now you sound like an investor :) My own vision is to take a queue that is relatively dumb and make it smarter. I want it to be able to, for example, allow you to rate limit workers without needing to implement this client side. And so on for all of the other bits that one needs to implement in the course of distributed processing. I’m still figuring out the market. Very large firms spending thousands of queues? Or developers who want a one stop solution built on familiar tech? Or hosting companies who want to offer their own queue as a service?
davidthewatson 2y ago
The whole idea here is superlative. +1 for k8s, kubernetes, cloud native, self-hosted, edge-enabled at low cost, no cost. I ran rq and minio for years on k8s, but been watching sqlite as a drop-in-replacement since most of my work has been early stage at or near the edge. Private cloud matters. This is an enabler. We've done too much already in public cloud where many things don't belong. BTLE sensors are perfectly happy talking to my Apple Watch directly with enough debugging. I'd argue the trip through cloud was not a win and should be corrected in the next generation of tools like this, where mobile is already well-primed for SQLite.
- memset 2y ago
  Really interesting. Question: when it comes to running these software on k8s, do you prefer to manage and host yourself, or do you use managed solutions on top of your own infra? (Do you pay for minio support?) Asking from a business perspective - I of course intend to keep developing this, but am also really trying to think through the business case as well.
  - davidthewatson 2y ago
    Good question! The answer depends on funding, i.e. in my own never-leaves-my-house case it is always self-host, much like SOC work. In the case of startup or research lab work (day-job, for lack of a better descriptor). It's frequently a slice of AWS, GCP, or Azure, i.e. 6 figure/mo cloud bills. I think those two broad cases are worth considering.
  - cpill 2y ago
    k3s for on premises deployments. I'm also using it for local self hosting side projects.
    - davidthewatson 2y ago
      Definitely a fan of k3s as k8s lite - complexity that your particular project may not need, particularly since every project doesn't need k8s and many are better off with less. If you look at the history from J2EE to the k8s prototype in Java to what we have now, it's a great idea to encapsulate all of these things into a single container, particularly at Google scale, but many unintended consequences arise from complexity accruing to features and functions which weren't actually requirements for your particular project being supported, i.e. the notion that YAGNI because few orgs have Google scale problems. If so, great! Carry on... If not, consider k3s or aptible or more emergent platforms I haven't actually used. The mere presence of unneeded items in source and documentation presents a "why am I here?" choice paradox. That's before we even get into keeping track of deprecations in the never-at-rest source/release evolution. Federation is a good example. I've worked at places that needed it and places that didn't.
skeeter2020 2y ago
I love that we're seeing a lot of projects apply the KISS principal and leverage (or take inspiration) from SQLite, like this, PocketBase and even DuckDB. An entire generation of developers (including myself) were tricked into thinking you had to build for scale from day one, or worse, took the path of least resistance right to the most expensive place for cloud services: the middle. I'm hopeful the next generation will have their introduction to building apps with simple, easy to manage & deploy stacks. The more I learn about SQLite, the more I love it.
- deleted 2y ago
  [deleted]
- memset 2y ago
  Thank you for the kind words! I wanted to get something out even if it doesn't have the scalability stuff built. But I have been thinking and architecting that behind the scenes, and now I "only" need to turn it into code. I think the queue itself will end up using a number of technologies: SQLite for some data (ie, organizing messages by date and queue), RocksDB for other things (fast lookup for messages), DuckDB (for message statistics and metadata), and other data structures. I find that some of the most performant software often uses a mix of different data structures and algorithms by introspecting the nature of the data it has. And if I can make one node really sing, then I can have confidence that distributing that will yield results. I think SQLite is a great start, but I really do want the software to be able to utilize the full resources the underlying hardware.
- jmspring 2y ago
  "build for scale", it really depends what needs to scale and what is "scale". Nearly every interview, every company these days thinks they need google or such level scaling. Nearly all do not.

656 points by memset 2y ago

localfirst 2y ago

[flagged]

pvg 2y ago
The purpose of Show HN is not really ‘product catalogue’
- localfirst 2y ago
  i didnt ask for a product catalogue, all im asking is to clarify whether you are truly FOSS or not by being transparent about the license. Any type of licensing conditions like AGPL 3.0 to force you to share your code is a no go for many many companies. I understand the desire to make money and you can do that without AGPL 3.0
  - bumbledraven 2y ago
    It's right there in the LICENSE file: https://github.com/poundifdef/SmoothMQ/blob/main/LICENSE https://github.com/poundifdef/SmoothMQ/blob/main/LICENSE What's unclear or not transparent about that?
    - localfirst 2y ago
      All I'm asking is to include the type of license in the Show HN title I had no idea there would be so much push back against this.
      - ncruces 2y ago
        I actually agree. The AGPL, for something like this, is a total non-starter, IMO, and just changes the conversation. It'd be one thing to choose the AGPL because you believe in the FOSS movement. Or if you're releasing something end-users can benefit from, directly, by self hosting it. To release something that will always be a component of something bigger, license it as AGPL, then talk about monetization… just cut the chase and release it as “source available,” with a licence people can actually use, even if it has a bunch of not-really-FOSS strings attached. Because who can realistically use this as is? Who can download the source and actually do something with it? What AGPL-compatible FOSS project is dying to use a drop-in AWS SQS replacement?
        localfirst 2y ago
        Just from IP/legal point of view AGPL and AGPL adjacent licensing that imposes control over how you use the code is a non-starter for many companies especially ones that are "backed by YC" proudly displayed on their website because that signals money will be exchanged in the future by design. MIT/BSD license that imposes no control over the how the code is used is the only FOSS we are allowed to use and we do donate regularly to maintainers working selflessly. What makes me angry is that somebody then uses that MIT/BSD license code, makes some modifications and releases it as AGPL with clear intent to make a buck from it without paying that original maintainer any money.
deleted 2y ago
[deleted]

philippta 2y ago

One quick suggestion on project structure: Move all the structs from models/ into the root directory. This allow users of this package to have nice and short names like: q.Message and q.Queue, and avoids import naming conflicts if the user has its own „models“ package.

memset 2y ago
Thanks for the tip - and for even looking at the code! I always struggle to figure out how to organize things.
- philippta 2y ago
  Just noticed that your root directory is already „package main“ so you can either move that to /cmd/something/ or simply rename models/ to q/. That would have the same effect and is also idiomatic.
pqb 2y ago
Actually, the "model/" (note: singular form) directory (package model) would be preferred in the golang world.

mannyv 2y ago

So I assume it does the back-end as well? I never cared to figure out what parts of SQS are clients-side and server side, but - does SmoothMQ support long polling, batch delivery, visibility timeouts, error handling, and - triggers? Or are triggers left to whatever is implementing the queue? Both FiFo and simple queues? Do you have throughput numbers? As an SQS user, a table of SQS features vs SmoothMQ would be handy. If it's just an API-compatible front-end then that would be good to know. But if it does more that would also be good to know. The reason you'd use this is because there are lots of clients who still want on-prem solutions (go figure). Being able to switch targets this way would be handy.

memset 2y ago
Everything here is on the backend. The client does very little except make api calls. It implements many of these features so far (ie, visibility timeouts) and there are some that are still in progress (long polling.) a compatibility table is a good idea.

deskr 2y ago

Perhaps do a small example application. go get github.com/poundifdef/SmoothMQ/models go: github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60 requires go >= 1.22.2; switching to go1.22.4 go: github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60 (matching github.com/poundifdef/SmoothMQ/models@upgrade) requires github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60: parsing go.mod: module declares its path as: q but was required as: github.com/poundifdef/SmoothMQ

memset 2y ago
Good idea! fyi this is not meant to be used as a library. It runs as a standalone server, and then your application connects to it using an existing AWS sdk.

xchip 2y ago

Thanks! Loving all these self-hosted KISS stuff :)

encoderer 2y ago

I think it’s better to say it’s sqs compatible versus an sqs replacement.

pqdbr 2y ago

How would one use this to replace Sidekiq for instance? Rate limiting is something I miss on Sidekiq (only available on the premium plan that I can't afford) and the gems that extend it break compatibility often.

koito17 2y ago

I may be asking a naive question, but what is the rationale behind disabling foreign key support and using them anyway in the database schema? See https://github.com/poundifdef/SmoothMQ/blob/46f8b22/queue/sqlite/sqlite.go#L56-L88 https://github.com/poundifdef/SmoothMQ/blob/46f8b22/queue/sq... The "TODO: check for errors" comment, combined with what seems like disabling foreign key constraint checks, makes me a bit hesitant to try this out.

memset 2y ago
Good question. It was an evolution. Originally I enforced foreign keys, but inserts/updates were unbearably slow. So I updated the connection string to disable them (but, as you point out, I haven't updated the CREATE TABLE statements.) In practice, I did not find they were necessary - I was only using foreign keys to automatically CASCADE deletes when messages were removed. But instead of relying on sqlite to do that, I do it myself and wrap the delete statements in transactions. There are many TODOs and error checkings that I will, over time, clean up and remove. I'm glad you've pointed them out - that's the great thing about open source, you at least know what you're getting into and can help shine a light on things to improve!
- plasma 2y ago
  Some of the slowdown will come from not indexing the FK columns themselves, as they need to be searched during updates / deletes to check the constraints.

john-tells-all 2y ago

Why not use LocalStack? It has SQS and a lot of AWS services for testing/development. Well documented, open source. https://docs.localstack.cloud/overview/ https://docs.localstack.cloud/overview/

crazysim 2y ago
I think the vibe might be that this is something someone could use for production.
- mtndew4brkfst 2y ago
  With no distribution model and built on SQLite, it certainly doesn't scream "production candidate" to me.
  - ku1ik 2y ago
    There’s more production systems on this planet running on a single node than ones running in a distributed setup. And SQLite is a rock solid database, deployed in more places than Postgres. The only thing here that could be taken when weighing production readiness would be the maturity and stability of this project itself.
  - gkbrk 2y ago
    You know there are millions of production deployments of SQLite right? It's perhaps the most widely deployed database.
- mannyv 2y ago
  "Production" is a big word to use for a piece of software. There are different levels of "production ready," depending on what you care about. Some enterprise production questions: * Is it HA? What HA topology is it designed for? * How many messages-per-second can it ingest? * What about with multiple clients? * Can you front-end it via a load balancer? * Can it guarantee only-one delivery? * How does it handle errors? ie: when a message is retrieved by a client and the client errors after the server dies, what happens to the message - does it redrive or expire correctly when the server comes back up? * How does it guarantee consistency in a multi-node environment? * How dies it behave when a node dies/runs out of memory? * How do you monitor it? * How do you recover a corrupted instance? * what happens if/when the SQLite partition runs out of space? etc etc. I use SQS as a main pipeline, and I would want to know a bunch of these things before I'd even consider replacing it. But for someone with one box on a VPS somewhere, you probably only care if it runs for a year without crashing

neuroscisoft 2y ago

This is very interesting! The self-hosted aspect is something I'll have to consider for certain purposes. My lab also developed an SQS-esque system based on the filesystem, so no dependencies whatsoever and no need for any operational system other than the OS. It doesn't support all SQS commands (because we haven't needed them), but it also supports commands that SQS doesn't have (like release all messages to visible status). https://github.com/seung-lab/python-task-queue https://github.com/seung-lab/python-task-queue

threecheese 2y ago

There is a lot of new ecosystem being built around SQLite in the browser using wasm, as a primary data store or as a local replica for every client, and there are some interesting interactions with crdt and peer to peer applications; does this fit into your business case? Would be interesting to see a massively distributed - via browser embedding - queue, that even uses standard sqs bindings.

andrewstuart 2y ago

I wrote sasquatch which is similar. sasquatch is more simple than SQS and queues are virtual (i.e the queue exists because you put a message in it, and is gone when there's no messages in it). https://github.com/crowdwave/sasquatch https://github.com/crowdwave/sasquatch sasquatch is also a message queue, also written in Golang and also based on sqlite. sasquatch implements behaviour very similar to SQS but does not attempt to be a dropin replacement. sqsquatch is not a complete project though, nor even really a prototype, just early code. Likely it does not compile. HOWEVER - sasquatch is MIT license (versus this project which is AGPL) so you are free to do with it as you choose. sasquatch is a single file of 700 lines so easy to get your head around: https://raw.githubusercontent.com/crowdwave/sasquatch/main/sasquatch.go https://raw.githubusercontent.com/crowdwave/sasquatch/main/s... Just remember as I say it's early code, won't even compile yet but functionally should be complete.

PanMan 2y ago

Correct me if I’m wrong but: SQLite sounds like: runs on one server. While that will work most of the time, it won’t work 100% of the time. I don’t know the specifics, but I’m fairly sure if a queue server crashes, SQS will keep working, as stuff is redundant. So while it can work in a best case, this (probably) won’t have the same reliability as SQS has..

memset 2y ago
I break it down two ways. First, the project does not yet have a distributed implementation, you’re correct. Stay tuned! Second, SQLite is incidental. Data is still stored on disk, just as SQS must be, but I’ve chosen SQLite as the file format for now.
- 22c 2y ago
  Strictly speaking, SQS does not necessarily store data on disk and being highly available and fault tolerant does not exclude one from operating a completely volatile data store. Indeed, SQLite itself has an in-memory option which, when appropriately architected, could be used as the primary data store for a highly available and fault tolerant queue. Having said that, I think it is safe to assume that SQS probably stores information to non-volatile storage. Nice work on the project!
parhamn 2y ago
The choice of SQLite is a great one as the distributed sqlite space is also growing. You have things like rqlite (raft+sqlite), and you have cloudservices like CloudFlare D1 which is basically distributed/HA sqlite. Plus you can swap it for basically any other sql database.
jpc0 2y ago
Does this even need to compete feature wise with SQS? If it faithfully reproduces the SQS Api what could possibly stop me from using this product now (if he ever does hosted) and then switching to SQS if the scale is ever justified? I'm all for a full suite of solutions targeting the same API. Can I run this thing on a single server with my dashboard app for the 3 person team I'm developing for and all that it cost them is per hour for tech support and server hardware and deloy my multi million user app on SQS but have effectively the same code.

ahachete 2y ago

Congratulations! I also love writing AWS API-compatible services. That's why I did Dyna53 [1] ;P (I know, unrelated, but hopefully funny) [1] https://dyna53.io https://dyna53.io

memset 2y ago
This is really fun!
selcuka 2y ago
It is fun, indeed. Could be a candidate for the best technology abuse of the year.
bensmoif 2y ago
This is an absolutely unhinged project idea. Great work

dav43 2y ago

Tangential but I looked at the codebase - because I like to imagine I can code (I can’t) - and for a laymen, I could follow the code. Makes me think quite positively of go lang and of the dev for designing in such a way. Can understand why teams like it because of easier maintenance Quite elegant from my uneducated pov.

memset 2y ago
Thank you for the kind words! I'll be sure to work on making it as complicated as I can :)
deleted 2y ago
[deleted]
x-complexity 2y ago
Honestly shows the capabilities of both: - The dev for making it both fast & simple to understand - Golang for making the codebase easy to follow
silisili 2y ago
That's the beauty of Go, generally, and why I immediately became attracted to it. The code is supremely readable because it enforces a style guide, and doesn't allow much magic. A number of people don't like it for limiting their expression and abilities, which I understand that feeling too. But as a middle aged programmer I realized that readability trumps conciseness and cleverness in the long run.
- bilekas 2y ago
  I can't remember who said it but I remember hearing that a good go Dev and a new go Dev should be able to produce generally similar code in order for maintainability. I do like playing with go, it's sits in a nice little playground in my head, mostly thanks to the compile times I think.
  - ayuhito 2y ago
    Two experienced Go devs tackle the same task, and their code will look almost identical. Two experienced Rust devs tackle the same task, and their solutions will be worlds apart. (I write both, but I do love Go for its simpleness)

leetrout 2y ago

Glad to find out what you've been up to! Very cool and I am excited to see where you take this.

dracarys18 2y ago

The only reason I use SQS is because how it's compatible with other AWS Services, I can have an S3 bucket whenever there's an upload I can send an SQS Message and I can have it trigger a lambda.

TheAnkurTyagi 2y ago

sounds like a great project. How do you handle performance with large message sizes?

throwaway984393 2y ago

[dead]

michaelcampbell 2y ago

This is really pedantic, totally outside any actual functionality, and may be a me thing, but a trigger for me is seeing left-justified (or worse, centered), columns of numbers (eg: your number of messages column). I'll grant it's small to infinitesimal, but you asked for feedback.

memset 2y ago
Hadn't even thought about that. Now I can't unsee it. Thanks for the suggestion.

lovestaco 2y ago

I was thinking of using Redis Stream instead of SQS, what would be better Redis or SmoothMQ?

tjoff 2y ago

SQS: Amazon Simple Queue Service

cwbrandsma 2y ago
Thank you, I was scratching my head trying to figure that out.
- rmbyrro 2y ago
  Why some people think we need AI everywhere to "augment" and "enrich" our "experiences". People don't bother a google search, after, what, 20 years in town?
deleted 2y ago
[deleted]

ninjazee124 2y ago

[flagged]

memset 2y ago
MongoDB is web scale :) https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
- simonw 2y ago
  Have you run any benchmarks yet? I'm guessing it happily handles thousands of queue operations a second given that it's SQLite and Go - would be interesting to see how settings like SQLite WAL mode affect its performance.
  - memset 2y ago
    I'm using WAL mode - I'd almost given up on sqlite until I remembered to enable it. I've been playing with benchmarks, yes! I haven't written them up yet because I worry about doing them the "right" way. but since you asked, I did a quick test with a single server and single client on a t2.nano. With 3 sending threads and 5 receiving threads, and mesg size of 2kb, I can send 700 msgs/s and receiver 500 msgs/s. It is way faster on my laptop, and I have a lot of tricks that I've been playing with to improve this.
    - withinboredom 2y ago
      I had to build a queue implementation not too long ago. We used t2.nano for our benchmarks as well. Your results are in-line with ours pre-optimizations (except we are using nats jetstream for queuing). I’d also recommend reducing the number of threads as that can increase performance on single-processor machines (context switching will kill you). Try to find the sweet spot (for us, it was 2x-4x the number of cpus to threads, at least, for our workload). If you are using go, it kinda sucks at single-cpu things, and if you detect that you have a single core, you can lock a go-routine to a thread and that sometimes helps. Also, try to batch sends (aka, for-loop) to attempt to saturate the send-side so by the time you come back with your next batch they’re still messages sitting in a network buffer somewhere. For example, we have a channel that wakes up our sender, then we honest-to-god wait like 60ms in the hopes there will be more than one message to pick up — and there usually is in production.

jerrygenser 2y ago

> In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues. are you monetizing this as a separate business from: https://www.ycombinator.com/companies/scratch-data https://www.ycombinator.com/companies/scratch-data

memset 2y ago
Truthfully, I don't know yet - I haven't even built a paid/hosted version at all. It is related to my existing business in the sense that it deals with realtime data. But I started working this as something I wish existed as opposed to having some big VC strategy and pitch deck behind it. (Also, I appreciate all of your feedback on this a month ago! It was really helpful to encourage me to keep looking into this and also figuring out the "first" things to launch with!)
- Kwpolska 2y ago
  Not every open source project needs to be monetized. Projects that were created to "scratch one's itch" tend to fare better than those built to make money. Devs put more love and less stress into them than into things they want to build a business out of. The monetization paragraph reads really weird, as if you believe HN is a community of VC-adjacent people looking for new ways to make money (it isn't), and talking about how you plan to exploit your new project is mandatory (it isn't either).
  - throwaway984393 2y ago
    [dead]
  - fragmede 2y ago
    > HN is a community of VC-adjacent people looking for new ways to make money it kinda is. There are all sorts of people here, but HN is owned by YC, a well known VC fund. That doesn't mean that everyone here is one, but it certainly influences the community here. > and talking about how you plan to exploit your new project is mandatory it's not mandatory but it's a frequently asked question. OP might as well answer it while they have the mic.

sgarland 2y ago

This looks great! How were you planning on tackling distributed?

memset 2y ago
This is a great question and something I've been thinking about a lot. Big picture: Each queue node can operate (mostly) independently, and this is good. As a consumer, I don't really care where my next message comes from, so I can minimize the amount of data that needs to have a "leader". The only data that needs to be synced is the list of queues, which doesn't change often. If one server is full, it should be able to route a request to another server. When we downscale, we can use S3/Dynamo (GCS/firestore) to store items and redistribute. There's more nitty gritty here (what about FIFO queues? What about replication?) but the fact that the main actions, "enqueue" and "dequeue", don't require lots of coordination makes this easier to reason about comapred to a full RDBMS.
- flaminHotSpeedo 2y ago
  You're hand-waving away all the complexity in the "nitty gritty." Enqueue absolutely requires coordination, if not via leader then at least amongst multiple nodes, if you want to guarantee at least once delivery If you don't guarantee that, cool, but you're not competing with sqs
- avinassh 2y ago
  Have you considered libsql? It is a distributed SQLite database, written mostly in Rust. There is a go database/sql and GORM driver available - https://github.com/tursodatabase/libsql https://github.com/tursodatabase/libsql disclaimer: I am one of the maintainers
  - memset 2y ago
    I did not know about libsql! Thanks for sharing. How do you think I should reason about making this distributed - should I use one of the (various) sqlite replicatoin libraries? Or should this be something I roll on my own (on top of some other protocol, such as raft, or something else.)

hrisen 2y ago

Keeping questions from scale & benchmarks aside, this is a cool thing for functional/unit testing module that uses SQS, instead of dumb mocks.

memset 2y ago
Thank you! Yes, you should definitely use this instead of localstack :)
- thrauat 2y ago
  Although I wouldn't recommend using LocalStack's SQS implementation for production workloads, calling it a dumb mock is a bit outdated. It emulates pretty much every SQS behvaior including long polling, delayed messages, visibility timeouts, DLQ redrive, batch send/receive, ...

okr 2y ago

A testcontainer could be useful. I use always this adobe-mock from s3 for my test. Hmm.

memset 2y ago
I was not aware of testcontainer until you just mentioned it! I have a very basic Dockerfile for deploying to fly.io. What/how would I get started? It looks like localstack is a supported testcontainer and they do support SQS (but I haven't tried it myself.)
- okr 2y ago
  Hmm, i never provided a testcontainer myself, but i have used docker images in a testcontainer. That is pretty straight-forward. The specialized testcontainers just pass down or expose more config parameters, imho, because they possess more knowledge about the docker container that it is about to start. For SQS i could imagine, that it is convenient to expose uri or arn or even auth parameters, that one then can refer to when setting up tests with dynamic config parameters and to setup the sqs client. Localstack i have not used so far. I use containers often for database tests also.

fbdab103 2y ago

Minor thing - the gif browser demo should have been maximized. The video zooming into the action on what would have fit onto a single screen was distracting. Are there maintenance actions that the admin needs to perform on the database? How are those done?

memset 2y ago
Thank you for the feedback - agree on the gif. Also it should probably show real messages an an example instead of rand(). re: maintenance - I have tried to build this to be hands-off. The only storage this uses is SQLite, and I have the code set to automatically vacuum as space increases. It also has a /metrics endpoint which has disk size. This is going to be used for two things in the future: first, as a metric for autoscaling (scale when disk is full) and second so that a server can stop serving requests when its disk is full (to prevent catastrophic failure.)

systemz 2y ago

Very cool. I needed something like this. Looking at short video in readme - I suspect adding live stats similar to sidekiq would make UI look more dynamic and allow quick diagnostics. Docs for current features are more important though.

memset 2y ago
Can you tell me more about why you needed this (and what you ended up using?) Totally agree on adding more metrics and information to the UI - but how much of that should be on the dashboard vs exposed as a prometheus metric for someone to use their dashboard tool of choice? I am not very good at visual design and have chosen the simplest possible tech to build it (static rendering + fomanticUI). I sometimes wonder if the lack of react or tailwind or truly beautiful elements will hold the project back.

stuaxo 2y ago

Good, we need open implementations of all the AWS stuff. I swear they reimplement stuff we have just so there are more places to bill us.

paulddraper 2y ago
SQS is cheap. 10m requests for $4. You will need hundreds of millions of requests per month for it to be noticable. And can an implementation like this even help you at that point?
- memset 2y ago
  Question: is there any feature set (in contrast to pricing) that would make it worthwhile for someone to switch?
  - threecheese 2y ago
    Operations. At my $bigCo we are always rolling our own ops tooling for queues, as it’s more than just “reprocess message”; there is always some business logic that separates different types of consumers/destinations, and so q management is never straightforward. Might be a differentiable feature.
    - memset 2y ago
      Can you tell me more about the types of queue things you need to implement? (Feel free to email me too - I’d be grateful for the feedback!)
      - threecheese 2y ago
        I’ll give you a real example, and it aligns with one of your feature goals that you mentioned elsewhere in these threads: allow you to rate limit without change on the client side. I have a workload that processes transactions for P external partners (P>1000) in a market with diverse technology sets; some partners are aggregated behind MSPs and some we integrate with directly, and so those P partners represent say 500 unique destinations. This workload handles < 100 line-of-business transaction types, spread across the partners; some support them all, some support only a few, and their implementations may be standard/consistent across transactions or their may be unique implementations. This is not a strange architecture, it represents the fact that every business-system has a lifecycle, there are legacy xml cruft and shiny graphql both exposed to the world and sitting next to each other, making money, and nearly every integrator has this. These systems go down, or networks become unavailable, or they perform maintenance, or maybe they can only support 2 concurrent connections - and every partner is different. Some MSPs come in and get some % of the market for handling this for businesses that don’t have the capability, but there are many that do. If you were in the business of exposing products that use these services, would you want your engineers having to care about any of it? Me neither, and so it is CRITICAL that we operationalize it, which for a store-and-forward integration workload basically ends up being an operational tool for queues and their destinations. I do not however find any tools that meet our requirements, which are simply “allow an operations/support team to manage transaction T for partner P who may be hiding behind MSP M” which includes rate limiting, circuit breaker, observability, routing, deployment, billing etc etc. This is IMO tied very closely to the queue - it is in fact the workload’s primary data store - and I do not understand why queue providers don’t differentiate their offerings with this kind of management capability. If I think about a business journey from startup to enterprise, maybe a small shop does not need this - they only support a small number of transactions and a small number of partners. As they grow, this requires engineering, and so they just build the tools they need to scale. But it doesn’t have to be that way.

deleted 2y ago

[deleted]

yimpolo 2y ago

This is super cool! I love projects that aim to create simple self-hostable alternatives to popular services. I assume this would work without much issue with Litestream, though I'm curious if you've already tried it. This would make a great ephemeral queue system without having to worry about coordinating backend storage.

memset 2y ago
I haven't tried this with litestream! It will be worth exploring that as a replication strategy. The nice thing about queues is that backend storage doesn't really need to be coordinated. Like, you could have two servers, with two sets of messages, and the client can just pull from them round robin. They (mostly) don't need to coordinate at all for this to work. However, this is different for replication where we have multiple nodes as backups.
- xrd 2y ago
  Since this is a single go binary I bet it would work perfectly with litestream. I use litestream in the exact same way with Pocketbase and it works perfectly (Pocketbase is also a single go binary). I'm going to give it a try and report back.

12_throw_away 2y ago

Actually pretty excited to try this, there are so many cases where I need a bare-bones (aka "simple"), local, persistent queue, but all the usual suspects (amqp, apache whatever, cloud nonsense, etc.) are way too heavy. I'll probably try poking at it directly through the HTTP API rather than an SDK ... does it need AWS V4 auth signatures or anything?

memset 2y ago
The only HTTP API that is exposed is actually the SQS one. (I'm not opposed to a "regular" HTTP API but the goal was to make it easy for people to use existing libraries.) If you do use your language's AWS SDK, the code handles [1] all of the V4 auth stuff. https://github.com/poundifdef/SmoothMQ/blob/main/protocols/sqs/sigv4.go https://github.com/poundifdef/SmoothMQ/blob/main/protocols/s... I'd love your feedback! Particularly the difficulties you find in running it, of which I'm sure there are many, so I can fix them or update docs.
stackskipton 2y ago
I'd also give a shout out to beanstalkd. I use it to teach message systems and it's great.

4RealFreedom 2y ago

The goals are a little different but I think it's worth pointing out ElasticMQ. I use it simulate sqs in a docker environment. https://github.com/softwaremill/elasticmq https://github.com/softwaremill/elasticmq

cies 2y ago
We use ElasticMQ to have an SQS compatible service for local development. We use it with docker-compose locally. In our remote envs we use SQS. So far I had not problems with ElasticMQ. I'm much intrigued by the small LOC count of SmoothMQ. When I compare it to ElasticMQ it's much smaller (probably by using sqlite's features). https://github.com/softwaremill/elasticmq https://github.com/softwaremill/elasticmq
memset 2y ago
I have looked at elasticmq but not played with it myself. You might also be interested in their benchmarks of all of the existing queues out there: https://softwaremill.com/mqperf/ https://softwaremill.com/mqperf/
- julius 2y ago
  I also just used ElasticMQ in a docker environment. Did you see any specific downsides, when you looked at it? In which scenarios should I consider replacing it with your solution?

chromanoid 2y ago

Could you elaborate why you didn't choose e.g. RabbitMQ? I mean you talk about AMQP and such, it seems to me, that a client-side abstraction would be much more efficient in providing an exit strategy for SQS than creating an SQS "compatible" broker. For example in the Java ecosystem there is https://smallrye.io/smallrye-reactive-messaging/latest/ https://smallrye.io/smallrye-reactive-messaging/latest/ that serves a similar purpose.

westurner 2y ago
Where AWS is the likely migration path for an app if it needs to be scaled beyond dev containers, already having tested with an SQS workalike prevents rework. The celery Backends and Brokers docs compare SQS and RabbitMQ AMQP: https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/index.html#broker-overview https://docs.celeryq.dev/en/stable/getting-started/backends-... Celery's flower utility doesn't work with SQS or GCP's {Cloud Tasks, Cloud Pub/Sub, Firebase Cloud Messaging FWIU} but does work with AMQP, which is a reliable messaging protocol. RabbitMQ is backed by mnesia, an Erlang/OTP library for distributed Durable data storage. Mnesia: https://en.wikipedia.org/wiki/Mnesia https://en.wikipedia.org/wiki/Mnesia SQLite is written in C and has lots of tests because aerospace IIUC. There are many extensions of SQLite; rqlite, cr-sqlite, postlite, electricsql, sqledge, and also WASM: sqlite-wasm, sqlite-wasm-http celery/kombu > Transport brokers support / comparison table: https://github.com/celery/kombu?tab=readme-ov-file#transport-comparison https://github.com/celery/kombu?tab=readme-ov-file#transport... Kombu has supported Apache Kafka since 2022, but celery doesn't yet support Kafka: https://github.com/celery/celery/issues/7674#issuecomment-1204053595 https://github.com/celery/celery/issues/7674#issuecomment-12...
- chromanoid 2y ago
  I don't get where you are pointing at. RabbitMQ and other MOMs like Kafka are very versatile. What is the use case for not using SQS right now, but maybe later? And if there is a use case (e.g. production-grade on-premise deployment), why not a client-side facade for a production-grade MOM (e.g. in Celery instead of sqs: amqp:)? Most MOMs should be more feature-rich than SQS. At-least-once delivery without pub/sub is usually the baseline and easy to configure. I mean if this project reaches its goal to provide an SQS compatible replacement, that is nice, but I wonder if such a maturity comes with the complexity this project originally wants to avoid.

rmbyrro 2y ago

I'd have named it MQLite. Congrats on finishing and delivering a project, this is quite a challenge in itself. I think SQLite can be a great alternative for many kinds of projects.

slotrans 2y ago

Not sure about the goal of providing a hosted service cheaper than SQS. SQS is already one of the cheapest services on Earth. It's pretty hard to spend more than a few bucks a month, even if you really try!

deleted 2y ago
[deleted]
PanMan 2y ago
This depends on your definition of few.. but SQS costs us more than the servers handling the messages… It’s cheap until you do billions of messages…
memset 2y ago
That is fair. Two things to validate from a biz perspective: 1. Is there some threshold where this would make sense financially (n billions of messages.) 2. Are the extra developer features (ie, larger message sizes, observability, DAGs) worth it for people to switch? Would love your thoughts - what, if anything, would make you even entertain moving to a different queue system?
- GordonS 2y ago
  Not the GP, but I also think you'll struggle to compete in the hosted service space. Maybe you could add a web admin GUI as a paid add on?
- threecheese 2y ago
  What is your target market? Cloud-native is, like the commenter said, going to be difficult to differentiate based on cost. I can see this being useful for hybrid cloud (onprem instances), functional or local testing, “localfirst” workloads, enthusiasts etc.
  - memset 2y ago
    Now you sound like an investor :) My own vision is to take a queue that is relatively dumb and make it smarter. I want it to be able to, for example, allow you to rate limit workers without needing to implement this client side. And so on for all of the other bits that one needs to implement in the course of distributed processing. I’m still figuring out the market. Very large firms spending thousands of queues? Or developers who want a one stop solution built on familiar tech? Or hosting companies who want to offer their own queue as a service?

davidthewatson 2y ago

The whole idea here is superlative. +1 for k8s, kubernetes, cloud native, self-hosted, edge-enabled at low cost, no cost. I ran rq and minio for years on k8s, but been watching sqlite as a drop-in-replacement since most of my work has been early stage at or near the edge. Private cloud matters. This is an enabler. We've done too much already in public cloud where many things don't belong. BTLE sensors are perfectly happy talking to my Apple Watch directly with enough debugging. I'd argue the trip through cloud was not a win and should be corrected in the next generation of tools like this, where mobile is already well-primed for SQLite.

memset 2y ago
Really interesting. Question: when it comes to running these software on k8s, do you prefer to manage and host yourself, or do you use managed solutions on top of your own infra? (Do you pay for minio support?) Asking from a business perspective - I of course intend to keep developing this, but am also really trying to think through the business case as well.
- davidthewatson 2y ago
  Good question! The answer depends on funding, i.e. in my own never-leaves-my-house case it is always self-host, much like SOC work. In the case of startup or research lab work (day-job, for lack of a better descriptor). It's frequently a slice of AWS, GCP, or Azure, i.e. 6 figure/mo cloud bills. I think those two broad cases are worth considering.
- cpill 2y ago
  k3s for on premises deployments. I'm also using it for local self hosting side projects.
  - davidthewatson 2y ago
    Definitely a fan of k3s as k8s lite - complexity that your particular project may not need, particularly since every project doesn't need k8s and many are better off with less. If you look at the history from J2EE to the k8s prototype in Java to what we have now, it's a great idea to encapsulate all of these things into a single container, particularly at Google scale, but many unintended consequences arise from complexity accruing to features and functions which weren't actually requirements for your particular project being supported, i.e. the notion that YAGNI because few orgs have Google scale problems. If so, great! Carry on... If not, consider k3s or aptible or more emergent platforms I haven't actually used. The mere presence of unneeded items in source and documentation presents a "why am I here?" choice paradox. That's before we even get into keeping track of deprecations in the never-at-rest source/release evolution. Federation is a good example. I've worked at places that needed it and places that didn't.

skeeter2020 2y ago

I love that we're seeing a lot of projects apply the KISS principal and leverage (or take inspiration) from SQLite, like this, PocketBase and even DuckDB. An entire generation of developers (including myself) were tricked into thinking you had to build for scale from day one, or worse, took the path of least resistance right to the most expensive place for cloud services: the middle. I'm hopeful the next generation will have their introduction to building apps with simple, easy to manage & deploy stacks. The more I learn about SQLite, the more I love it.

deleted 2y ago
[deleted]
memset 2y ago
Thank you for the kind words! I wanted to get something out even if it doesn't have the scalability stuff built. But I have been thinking and architecting that behind the scenes, and now I "only" need to turn it into code. I think the queue itself will end up using a number of technologies: SQLite for some data (ie, organizing messages by date and queue), RocksDB for other things (fast lookup for messages), DuckDB (for message statistics and metadata), and other data structures. I find that some of the most performant software often uses a mix of different data structures and algorithms by introspecting the nature of the data it has. And if I can make one node really sing, then I can have confidence that distributing that will yield results. I think SQLite is a great start, but I really do want the software to be able to utilize the full resources the underlying hardware.
jmspring 2y ago
"build for scale", it really depends what needs to scale and what is "scale". Nearly every interview, every company these days thinks they need google or such level scaling. Nearly all do not.