5 ms·
I prefer Celery for Python. Far more light-weight, and intended for distributed task processing. http://celery.readthedocs.org/ http://celery.readthedocs.org/
by narsil 11y ago
I prefer Celery for Python. Far more light-weight, and intended for distributed task processing.
http://celery.readthedocs.org/ http://celery.readthedocs.org/
- brianwawok 11y agoAre those really competing though? Celery rocks for some background image processing. Not for multinode analytics of huge data sets.
- throwaway-anon 11y agoActually that's not true. Have a look at the airflow project by AirBnB. https://github.com/airbnb/airflow https://github.com/airbnb/airflow
- squeaky-clean 11y agoDoesn't seem like it can handle processing on real-time streams of data. They don't really seem comparable. From the readme: "Airflow is not a data streaming solution. Tasks do not move data from one to the other (though tasks can exchange metadata!). Airflow is not in the Spark Streaming or Storm space, it is more comparable to Oozie or Azkaban."
- brianwawok 11y agoThat doesn't really seem the same at all.
- throwaway-anon 11y agoI was replying to a comment not saying airflow is capable of stream processing. > Not for multinode analytics of huge data sets. Airflow does this just fine with celery.