5 ms·

That looks very interesting indeed, and love to see some more development in that space. Would anyone care to explain how it differs from Airflow? I had dismis

That looks very interesting indeed, and love to see some more development in that space. Would anyone care to explain how it differs from Airflow? I had dismissed Airflow some months ago for my use case (Windows, and large number of dependencies were an issue at the time), but would still like to eventually migrate my ETL scripts to a solid framework sometime in the future.

stadeschuldt 8y ago
Mara is designed to be simpler than Airflow. It positions itself between plain scripts and Airflow.
- huac 8y ago
  What features does it forgo to be 'simpler'?
  - artwr 8y ago
    The main feature which jumps out at me, I think it schedules on a single node, where Airflow can orchestrate across multiple nodes. Airflow is also starting to implement more enterprise-y features, like role-based access control, which should be coming to the next release.
    - martin_loetzsch 8y ago
      (author here) That's absolutely correct. Mara uses Python's multiprocessing [1] to parallelize pipeline execution [2] on a single node so it doesn't need a distributed task queue. Beyond that (and visualization) it can't do much. In fact it doesn't even have a scheduler (you can use jenkins or cron for that). [1] https://docs.python.org/3.6/library/multiprocessing.html https://docs.python.org/3.6/library/multiprocessing.html [2] https://github.com/mara/data-integration/blob/master/data_integration/execution.py https://github.com/mara/data-integration/blob/master/data_in...
artwr 8y ago
With a little bit of work you could probably trigger tasks on a Windows Machine, in particular if the work you need to do is mostly script and/or SQL based. The main limit is that Gunicorn for the Airflow webserver is not currently compatible with Windows, but the scheduler should work fine. I believe it might be able to run in the Windows Subsystem for Linux, but I don't think anyone has tested it as of yet. (Source: I am an Apache Airflow committer)
- leblancfg 8y ago
  Thanks! To be fair, I was looking at it from a single-user perspective, and was optimizing for: time it takes to setup and move my ETL to Airflow + time I will save in the future by using the framework vs time I will spend fixing and maintaining my already-existing ETL scripts so I dropped the notion after I hit the first major snag. But I should put a reminder to take a look back at it in a few months!

stadeschuldt 8y ago

Mara is designed to be simpler than Airflow. It positions itself between plain scripts and Airflow.

huac 8y ago
What features does it forgo to be 'simpler'?
- artwr 8y ago
  The main feature which jumps out at me, I think it schedules on a single node, where Airflow can orchestrate across multiple nodes. Airflow is also starting to implement more enterprise-y features, like role-based access control, which should be coming to the next release.
  - martin_loetzsch 8y ago
    (author here) That's absolutely correct. Mara uses Python's multiprocessing [1] to parallelize pipeline execution [2] on a single node so it doesn't need a distributed task queue. Beyond that (and visualization) it can't do much. In fact it doesn't even have a scheduler (you can use jenkins or cron for that). [1] https://docs.python.org/3.6/library/multiprocessing.html https://docs.python.org/3.6/library/multiprocessing.html [2] https://github.com/mara/data-integration/blob/master/data_integration/execution.py https://github.com/mara/data-integration/blob/master/data_in...

artwr 8y ago

With a little bit of work you could probably trigger tasks on a Windows Machine, in particular if the work you need to do is mostly script and/or SQL based. The main limit is that Gunicorn for the Airflow webserver is not currently compatible with Windows, but the scheduler should work fine. I believe it might be able to run in the Windows Subsystem for Linux, but I don't think anyone has tested it as of yet. (Source: I am an Apache Airflow committer)

leblancfg 8y ago
Thanks! To be fair, I was looking at it from a single-user perspective, and was optimizing for: time it takes to setup and move my ETL to Airflow + time I will save in the future by using the framework vs time I will spend fixing and maintaining my already-existing ETL scripts so I dropped the notion after I hit the first major snag. But I should put a reminder to take a look back at it in a few months!