4 ms·
"correct" is a value judgement that depends on lots of different things. Only you can decide which tool is correct. Here are some ideas: - https://camel.apache
by lantry 3y ago
"correct" is a value judgement that depends on lots of different things. Only you can decide which tool is correct. Here are some ideas:
- https://camel.apache.org/ https://camel.apache.org/
- https://www.windmill.dev/ https://www.windmill.dev/
- https://github.com/huginn/huginn https://github.com/huginn/huginn
Your idea about a queue (in redis, or postgres, or sqlite, etc) is also totally valid. These off-the-shelf tools I listed probably wouldn't give you a huge advantage IMO.
- idorosen 3y agoI agree there are many options in this space. Two others to consider: - https://airflow.apache.org/ https://airflow.apache.org/ - https://github.com/spotify/luigi https://github.com/spotify/luigi There are also many Kubernetes based options out there. For the specific use case you specified, you might even consider a plain old Makefile and incrond if you expect these all to run on a single host and be triggered by a new file showing up in a directory… I like Airflow because you can give access to the web UI to operators and they can kick/run/stop tasks or graphs of tasks. Both Airflow and Luigi expect you to express your workflow as a DAG in Python code.
- vsroy 3y agoIt seems like the modern solution is to use something like temporal.io -- (related to windmill). Alas, surely people have been solving this problem for ages now without temporal.io