3 ms·

Good, we need open implementations of all the AWS stuff. I swear they reimplement stuff we have just so there are more places to bill us.

• paulddraper 2y ago
  SQS is cheap. 10m requests for $4. You will need hundreds of millions of requests per month for it to be noticable. And can an implementation like this even help you at that point?
  • memset 2y ago
    Question: is there any feature set (in contrast to pricing) that would make it worthwhile for someone to switch?
    • threecheese 2y ago
      Operations. At my $bigCo we are always rolling our own ops tooling for queues, as it’s more than just “reprocess message”; there is always some business logic that separates different types of consumers/destinations, and so q management is never straightforward. Might be a differentiable feature.
      • memset 2y ago
        Can you tell me more about the types of queue things you need to implement? (Feel free to email me too - I’d be grateful for the feedback!)
        • threecheese 2y ago
          I’ll give you a real example, and it aligns with one of your feature goals that you mentioned elsewhere in these threads: allow you to rate limit without change on the client side. I have a workload that processes transactions for P external partners (P>1000) in a market with diverse technology sets; some partners are aggregated behind MSPs and some we integrate with directly, and so those P partners represent say 500 unique destinations. This workload handles < 100 line-of-business transaction types, spread across the partners; some support them all, some support only a few, and their implementations may be standard/consistent across transactions or their may be unique implementations. This is not a strange architecture, it represents the fact that every business-system has a lifecycle, there are legacy xml cruft and shiny graphql both exposed to the world and sitting next to each other, making money, and nearly every integrator has this. These systems go down, or networks become unavailable, or they perform maintenance, or maybe they can only support 2 concurrent connections - and every partner is different. Some MSPs come in and get some % of the market for handling this for businesses that don’t have the capability, but there are many that do. If you were in the business of exposing products that use these services, would you want your engineers having to care about any of it? Me neither, and so it is CRITICAL that we operationalize it, which for a store-and-forward integration workload basically ends up being an operational tool for queues and their destinations. I do not however find any tools that meet our requirements, which are simply “allow an operations/support team to manage transaction T for partner P who may be hiding behind MSP M” which includes rate limiting, circuit breaker, observability, routing, deployment, billing etc etc. This is IMO tied very closely to the queue - it is in fact the workload’s primary data store - and I do not understand why queue providers don’t differentiate their offerings with this kind of management capability. If I think about a business journey from startup to enterprise, maybe a small shop does not need this - they only support a small number of transactions and a small number of partners. As they grow, this requires engineering, and so they just build the tools they need to scale. But it doesn’t have to be that way.
  • mlhpdx 2y ago
    Not just cheap per message, but queues themselves are free so there is zero “reserved” cost - it’s pay per message. Add to that, it’s enormously scalable in terms of throughput and retained messages. And it’s globally available. A single file app is no comparison, really. The value of SQS is in the engineering.
• memset 2y ago
  Curious: if you were going to switch, how would you want to run this? Would you want to deploy it to your own EC2 instances, or would you want a hosted solution (just as SQS itself is?)
  • stackskipton 2y ago
    Self hosted probably. Why buy knock off SQS in the cloud when real thing is right there? If you are greenfield and scared of hitching yourself to Amazon, why not go something like RabbitMQ? There is also RabbitMQ cloud providers as well.
    • jpalomaki 2y ago
      Unless you need the features, I would steer away from the "real message queues". Running these adds complexity, especially if you want higher availability and they are not always cheap. HTTP based solution is easy to understand and implemented (if basic functionality is enough for your use case). Real MQs obviously have more complex protocols and not all client libraries are perfect.
      • stackskipton 2y ago
        I find basics of most AMQP systems to be extremely reliable. Message on, one time delivery with message hiding until expiration of TTL. Sure, you can get into some insane setups with RabbitMQ/Kafka and that's where libraries features don't always match up. But SQS probably wouldn't work for you either in those cases.
  • nilamo 2y ago
    Ideally this would be run within a k8s cluster, to easily fit with the other services.