3 ms·
Instead of SQS it uses RabbitMQ. I don't know about it's limits but since a serious RabbitMQ installation can handle something like millions of messages per se
by timf 17y ago
Instead of SQS it uses RabbitMQ. I don't know about it's limits but since a serious RabbitMQ installation can handle something like millions of messages per second I assume the limits you will hit are in the persistence solution on celeryd. It can use a Tokyo Tyrant or MongoDB backend instead of a RDBMS, as well as memcached support. Those seem like they would help in that department.
I don't think you can change trees after they are sent (you mean subtasks, right?).
You can see logging example at "defining and executing tasks" section here: http://ask.github.com/celery/introduction.html#usage http://ask.github.com/celery/introduction.html#usage
Sorry, I am only a light user at this point in time.
- darrellsilver 17y agoI've been hearing really good things about RabbitMQ, and haven't been particularly impressed with SQS. It works at scale (we're currently at 10s of thousands a day, but it's probably the same performance at 100 or 1000x that...) but tasks take many seconds (like 10-30) to show up in the queue, which absolutely kills us. We'll be adding a RabbitMQ plugin to Norc, just like SQS, when we can.
- vegai 17y agohttp://www.rabbitmq.com/faq.html#performance http://www.rabbitmq.com/faq.html#performance "From our testing, we expect easily-achievable throughputs of 4000 persistent, non-transacted one-kilobyte messages per second (Intel Pentium D, 2.8GHz, dual core, gigabit ethernet) from a single RabbitMQ broker node writing to a single spindle." Or what do you mean by serious?
- timf 17y agoI was thinking more about multiple brokers working together. Also, hmm, I didn't think rabbit was so significantly behind other AMQP implementations like zeromq: "4,100,000 messages a second" - http://www.zeromq.org/ http://www.zeromq.org/
- asksol 17y agoJust a small note: You can also use CELERY_BACKEND="amqp" to send back the result as a message, it's the most efficient way, but then you can only look up the result once (unless you send another message with the same result).