4 ms·
Ask HN: What is the correct way to deal with pipelines?
I have an application where once a job succeeds it needs to go through a series of steps, some are CPU intensive (ffmpeg, etc.), others require communicating with an external API that may fail (Discord), etc.
This is not a massive data-engineering project. I just need to ensure the steps get run to completion.
My current plan to do this is to just use Redis lists as queues, and then have steps go from one queue to the next, but I'm wondering if there's a better way.
- lantry 3y ago"correct" is a value judgement that depends on lots of different things. Only you can decide which tool is correct. Here are some ideas: - https://camel.apache.org/ https://camel.apache.org/ - https://www.windmill.dev/ https://www.windmill.dev/ - https://github.com/huginn/huginn https://github.com/huginn/huginn Your idea about a queue (in redis, or postgres, or sqlite, etc) is also totally valid. These off-the-shelf tools I listed probably wouldn't give you a huge advantage IMO.
- idorosen 3y agoI agree there are many options in this space. Two others to consider: - https://airflow.apache.org/ https://airflow.apache.org/ - https://github.com/spotify/luigi https://github.com/spotify/luigi There are also many Kubernetes based options out there. For the specific use case you specified, you might even consider a plain old Makefile and incrond if you expect these all to run on a single host and be triggered by a new file showing up in a directory… I like Airflow because you can give access to the web UI to operators and they can kick/run/stop tasks or graphs of tasks. Both Airflow and Luigi expect you to express your workflow as a DAG in Python code.
- vsroy 3y agoIt seems like the modern solution is to use something like temporal.io -- (related to windmill). Alas, surely people have been solving this problem for ages now without temporal.io
- dylanhassinger 3y agoBuddy.works is my favorite CI tool