3 ms·
I'd not heard of airflow before. Some of the other comments mention using it for ETL jobs. The webpage makes it sound like a generic job scheduler. Jenkins i
by aidenn0 6y ago
I'd not heard of airflow before. Some of the other comments mention using it for ETL jobs. The webpage makes it sound like a generic job scheduler.
Jenkins is also a generic job scheduler, but it is primarily used for CI/CD. It seems like there's some overlap here though; I'm interested to hear other peoples thoughts on "I have a bunch of jobs that need to be run; they aren't necessarily ETL or CI/CD; what tool is good for this, and how would I differentiate between them."
It seems that mostly people use whatever they have setup, so if someone already does CI/CD through jenkins, they will add other unrelated jobs to jenkins just to avoid setting something else up. Any thoughts on if things are being left on the table here, as even generic tools can make you feel like you are "swimming up stream" if you are doing things that the rest of the community is not with them.
- pablobaz 6y agoI’ve used both extensively. Here is my personal opinion of some Pros and Cons for the types of jobs that are in the fuzzy area of overlapping functionality. Jenkins gui and interface handles lots of standalone jobs more nicely. Jenkins has much better support for parameterised jobs that can be kicked off manually. Airflow can handle dependencies between jobs in a much better way. Nicely defining and visualising dags of job really is the killer feature. Browsing task/job logs is nicer in Airflow IMO. Airflow scheduler is flaky - hopefully better in 2.0. Airflow has much more “magic” than Jenkins this is often infuriating. All that said, my preference is to move jobs to Airflow. Getting a nice gui for manually triggered jobs in airflow is/was the only large missing piece for me.
- mixmastamyk 6y ago> Getting a nice gui for manually triggered jobs… So there is a GUI, but it isn't nice. Am I reading that correctly? Or perhaps the functionality doesn't exist at all? Or can be done at the command line or API?
- shmoogy 6y agoI love airflow because it shows you the tasks in a tree view, and dag view, and you can trigger jobs manually. The UI is fine, but looks to be improved in 2.0. Main improvement I am looking forward to are the scheduler changes, I run a few jobs that trigger every minute in airflow, and it's definitely not intended for that.
- morelandjs 6y agoCheckout Prefect
- mr_toad 6y agoThe reason that scheduling and build tools tend to overlap in functionality is that they are both at heart tools for processing DAGs.