4 ms·
(author here) That's absolutely correct. Mara uses Python's multiprocessing [1] to parallelize pipeline execution [2] on a single node so it doesn't need a dis
by martin_loetzsch 8y ago
(author here)
That's absolutely correct. Mara uses Python's multiprocessing [1] to parallelize pipeline execution [2] on a single node so it doesn't need a distributed task queue. Beyond that (and visualization) it can't do much. In fact it doesn't even have a scheduler (you can use jenkins or cron for that).
[1] https://docs.python.org/3.6/library/multiprocessing.html https://docs.python.org/3.6/library/multiprocessing.html
[2] https://github.com/mara/data-integration/blob/master/data_integration/execution.py https://github.com/mara/data-integration/blob/master/data_in...