8 ms·
What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
by jszymborski 2y ago
What do people recommend?
Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
- giovannibonetti 2y agoRedis, SQLite or even a traditional DB like Postgres or MySQL can all do a better job than that.
- alexwebr 2y ago(Author here) RabbitMQ or AWS SQS are probably good choices.
- kgeist 2y agoWe use RabbitMQ, and workers simply pull whatever is next in the queue after they finish processing their previous jobs. I’ve never witnessed jobs piling up for a single consumer.
- sea-gold 2y agoNATS https://docs.nats.io/nats-concepts/overview/compare-nats https://docs.nats.io/nats-concepts/overview/compare-nats
- MuffinFlavored 2y agoNATS/WebSockets are good for 1 publisher -> many consumer (pubsub) RabbitMQ is good for 1 producer -> 1 consumer with ack/nack Right?
- esafak 2y agoNATS does many-to-many.
- Joel_Mckay 2y agoActually, I used RabbitMQ static routes to feed per-cpu-core single thread bound consumers that restart their process every k transactions, or watchdog process timeout after w seconds. This prevents cross contamination of memory spaces, and slow fragmentation when the parsers get hammered hard. RabbitMQ/Erlang on OTP is probably one of the most solid solutions I've deployed over the years (low service cycle demands.) Highly recommended with the AMQP SSL credential certs, and GUID approach to application layer load-balancing. Cut our operational costs around 37 times lower than traditional load-balancer approaches. =3
- relistan 2y agoAgree. RabbitMQ is a Swiss Army knife that has a lot of available patterns, scales fairly well, and is very robust. If you don’t know what to choose, start with Rabbit. It will let you figure out which patterns you need and you probably won’t scale out of it. Pay someone to host it. On the other hand, if you know what you need to do and it’s supported by it, NATS is IME the way to go (particularly JetStream).
- rcombatwombat 2y agoIMHO NATS is an even better Swiss army knife than Rabbit
- rcombatwombat 2y agoNATS does much more than pub/sub (with any fan out), it also does queueing in Core NATS (no persistence) and streaming including queuing over streams (with ack/nack/term). Don't be fooled into thinking NATS doesn't do queueing because it doesn't have a "Q" in the name :).
- rogerthis 2y agoI'd wish NATS were more popular. It feels it lacks some real big sponsors $$$.
- hhh 2y agoScaling with NATS seems weird. I like what i’ve seen with others using it though
- deleted 2y ago[deleted]
- stickfigure 2y agoUntil you hit scale, the database you're already using is fine. If that's Postgres, look up SELECT FOR UPDATE SKIP LOCKED. The major convenience here - aside from operational simplicity - is transactional task enqueueing. For hosted, SQS or Google Cloud Tasks. Google's approach is push-based (as opposed to pull-based) and is far and above easier to use than any other queueing system.
- vrosas 2y agoCloud Tasks is one of the most undervalued tools in the GCP ecosystem, but mostly because PubSub gets all the attention. I've been using it since it was baked in the AppEngine and love it for 1-to-1 queues or delayed job handling.
- kamikaz1k 2y agohow do you recommend working with Cloud Tasks? raw dogging gcloud? Terraform? or something more manageable? I've been curious for one of my smaller projects, but I am worried about adopting more GCPisms.
- vrosas 2y agoTerraform is definitely for the best. Any AI tool should be able to spit it out well enough, but if you do rawdog it in the console or gcloud you might be able to export the terraform with: gcloud beta resource-config bulk-export --resource-format=terraform
- stickfigure 2y agoBack when I was all-in on GCP, I had a queue.yaml file which the appengine deployer syncs to cloud tasks (creates/disabled queues, changes the rate limits, concurrency, etc). Now that I'm mostly on AWS... I still use the same system. I have a thin little project that deploys to GAE and has a queue.yaml file. It sets up the cloud tasks queues. They hit my EB endpoints just like they used to hit my GAE endpoints. As a bonus, my thin little GAE app also has a cron.yaml that it proxies to my AWS app. Appengine's cron is also better than Amazon's overcomplicated eventbridge system. It's great.
- ha-shine 2y agoI use Temporal (https://temporal.io/ https://temporal.io/)
- ozarker 2y agoSQS, Azure Service Bus, RabbitMQ, ActiveMQ, QPID, etc… any message broker that provides the competing consumer pattern. though I’ll say having managed many of these message brokers myself, it’s definitely better paying for a managed service. They’re a nightmare when you start running into problems.
- sanex 2y agoIf you're using .NET I have to plug https://particular.net/ https://particular.net/ Nservicebus from particular.net. It's great at abstracting away the underlying message broker and provides an opinionated way to build a distributed system.
- stackskipton 2y ago.Net SRE here, please no. Take 5 minutes to learn your messaging bus SDK and messaging system instead of yoloing some library that you don't understand. It's really not that hard. Also, ServiceControl, ServiceInsight and ServicePulse are inventions of developers who are clearly WinAdmins who don't know what modern DevOps is. If you want to use that, you are bad and should feel bad. (Sorry, I have absolute rage around this topic) EDIT: If you insist, use MassTransit (https://masstransit.io/ https://masstransit.io/)
- jahsome 2y agoAs a linux fanboy recently trapped in a windows world, I actually find Particular stuff not so bad to work with. It's on the friendlier end of the spectrum among the tooling I help manage, at least compared to Microsoft crap. Either way I'm feeling quite validated by your rage, so thanks for sharing. I feel like we could be good friends.
- amazingamazing 2y agoSqs- easy and cheap.
- lmm 2y agoKafka. If your load is low enough for the problem described in the article to happen, your load is low enough that it's not an issue.
- xmcqdpt2 2y agoI'm not sure you understood the article. You can have a very low load but each task on your queue takes a while to process, in which case you want fair distribution of work.
- lmm 2y agoThe distribution is fair - everything is round-robin, so in the long run each worker receives the same rate of tasks. It's just "lumpy" - sometimes you can get a big batch sent to one worker, then a big batch sent to another worker - but it will all average out.
- sc68cal 2y ago* Redis pub/sub * Redis streams * Redis lists (this is what Celery uses when Redis backend is configured) * RabbitMQ * ZeroMQ
- est 2y agoThis. If you have really small volume like this article describes, just use Redis.
- stackskipton 2y agoDatabase is good recommendation. Also give a shoutout to Beanstalkd (https://beanstalkd.github.io/ https://beanstalkd.github.io/)
- petepete 2y agoI love beanstalkd, used it for plenty over the years and it just flies through with no fuss. Plus, it's fully supported by Rails' ActiveJob.
- stephen_g 2y agoHas anyone used Redpanda? I stumbled upon it when researching streaming, it claims to be Kafka compatible but higher performance and easier to manage. Haven't tried it myself but interested if anyone else has experience.
- rockwotj 2y agoPlenty of people choose Redpanda because it’s the easiest getting started experience for kafka. There is a single binary for the full broker, where I have never seen Apache Kafka as easy to setup. It’s got a great UI as well. The quickstart to getting everything running locally all documented here: https://docs.redpanda.com/current/get-started/quick-start/#deploy-redpanda-self-managed https://docs.redpanda.com/current/get-started/quick-start/#d... Disclaimer: I work at Redpanda
- gnfargbl 2y agoPulsar. Works extremely well as both a job queue and a data bus. We have been using it in this application for half a decade now with no serious issues. I don't understand why it doesn't get more popular attention.
- akshayshah 2y agoEven StreamNative is effectively abandoning Pulsar and going all-in on the Kafka protocol. I can see the theoretical benefits of Pulsar, but it just doesn’t seem to have the ecosystem momentum to compete with the Kafka juggernaut.
- gnfargbl 2y agoThe advantages of Pulsar are very much practical, at least for us. Without it we would have to manage two separate messaging systems. I don't see any evidence of StreamNative abandoning Pulsar at this point. I do see a compatibility layer for the Kafka protocol. That's fine.
- akshayshah 2y agoIt sure looks like they’re going quite a ways beyond Kafka-on-Pulsar - the Ursa/Oxia work they’re focused on right now replaces BookKeeper and seems very firmly Kafka-oriented. Or does Ursa also work with the Pulsar protocol?
- matteomerli 2y agoUrsa supports Pulsar protocol as well
- VenturingVole 2y agoPulsar vs Kafka was a significant lesson to me: The "best" technology isn't always the winner. I put it in quotes because I'm a massive fan of Pulsar and addressing the shortcomings of Kafka. However, with regards to some choices at a former workplace: The broader existing support/integration ecosystem along with Confluent's commercial capabilities won out with regards to technology choices and I was forced to acquiesce. A bit like Betamax vs VHS, albeit that one pre-dates me significantly.
- kitd 2y agoKafka with a different partitioner would have worked fine. The problem was that the web workers loaded up the same partition. Randomising the chosen partition would have removed, or at least alleviated, the stated problem.
- jiaaro 2y agoRandom and round robin partitioning are the configurations being discussed. The main point of the article is that low message volumes mean you can get unlucky and end up with idle workers when there is still work to be done