3 ms·

What's the best pipeline/workflow management tool for command line programs with a GUI? E.g. for resuming the process after it gets interrupted etc.

c0l0 7y ago
I am not kidding: A graphical terminal emulator. Mastering the usual command line interface (terminal emulator, interactive shell, maybe a terminal multiplexer) is non-optional if you want to use CLI tools at or close to peak effectiveness.
- sansnomme 7y ago
  I mean task resumption after interruption etc. Like airflow type of tools. Not quite unix task suspend options, this is about data pipelines. For Hadoop-style MapReduce, you can split the task into jobs which can be resumed and discarded etc. Shell scripting is not an elegant way to deal with this, a proper orchestrator tool is better.
  - cure 7y ago
    You could try the tool my group builds, Arvados (https://arvados.org https://arvados.org). We use Common Workflow Language (CWL) as the workflow language. Arvados works great for very large computations and data management at petabyte scale. It really shines in a production environment where data provenance is key. Intelligent handling of failures (which are inevitable at scale) is a key part of the Arvados design.
samuell 7y ago
Pipeline mgmt tool with a GUI, or tool for cli tools with a GUI? ;) One very widely used option is Galaxy https://galaxyproject.org https://galaxyproject.org You might also want to check out: - Arvados https://arvados.org https://arvados.org - Chipster https://chipster.csc.fi https://chipster.csc.fi - Knime https://www.knime.com https://www.knime.com They each have their strenghts and weaknesses. I'd probably be going with Galaxy as the default option, unless you find specific reasons to go with some of the others.
dredmorbius 7y ago
Screen, if it's the connection session itself. Tmux if you're new-school. Task-splitting itself is inherently recoverable, as incomplete work units don't produce the final output you're looking for, and can. be retried, re-run, or re-entered into the task pool. A GUI quite frankly simply gets in the way.
rodgerd 7y ago
CMake has a GUI (https://www.johnlamp.net/cmake-tutorial-3-gui-tool.html https://www.johnlamp.net/cmake-tutorial-3-gui-tool.html), and Make is pretty much perfect for your use case.

c0l0 7y ago

I am not kidding: A graphical terminal emulator. Mastering the usual command line interface (terminal emulator, interactive shell, maybe a terminal multiplexer) is non-optional if you want to use CLI tools at or close to peak effectiveness.

sansnomme 7y ago
I mean task resumption after interruption etc. Like airflow type of tools. Not quite unix task suspend options, this is about data pipelines. For Hadoop-style MapReduce, you can split the task into jobs which can be resumed and discarded etc. Shell scripting is not an elegant way to deal with this, a proper orchestrator tool is better.
- cure 7y ago
  You could try the tool my group builds, Arvados (https://arvados.org https://arvados.org). We use Common Workflow Language (CWL) as the workflow language. Arvados works great for very large computations and data management at petabyte scale. It really shines in a production environment where data provenance is key. Intelligent handling of failures (which are inevitable at scale) is a key part of the Arvados design.

samuell 7y ago

Pipeline mgmt tool with a GUI, or tool for cli tools with a GUI? ;) One very widely used option is Galaxy https://galaxyproject.org https://galaxyproject.org You might also want to check out: - Arvados https://arvados.org https://arvados.org - Chipster https://chipster.csc.fi https://chipster.csc.fi - Knime https://www.knime.com https://www.knime.com They each have their strenghts and weaknesses. I'd probably be going with Galaxy as the default option, unless you find specific reasons to go with some of the others.

dredmorbius 7y ago

Screen, if it's the connection session itself. Tmux if you're new-school. Task-splitting itself is inherently recoverable, as incomplete work units don't produce the final output you're looking for, and can. be retried, re-run, or re-entered into the task pool. A GUI quite frankly simply gets in the way.

rodgerd 7y ago

CMake has a GUI (https://www.johnlamp.net/cmake-tutorial-3-gui-tool.html https://www.johnlamp.net/cmake-tutorial-3-gui-tool.html), and Make is pretty much perfect for your use case.