4 ms·
I am not kidding: A graphical terminal emulator. Mastering the usual command line interface (terminal emulator, interactive shell, maybe a terminal multiplexer
by c0l0 7y ago
I am not kidding: A graphical terminal emulator.
Mastering the usual command line interface (terminal emulator, interactive shell, maybe a terminal multiplexer) is non-optional if you want to use CLI tools at or close to peak effectiveness.
- sansnomme 7y agoI mean task resumption after interruption etc. Like airflow type of tools. Not quite unix task suspend options, this is about data pipelines. For Hadoop-style MapReduce, you can split the task into jobs which can be resumed and discarded etc. Shell scripting is not an elegant way to deal with this, a proper orchestrator tool is better.
- cure 7y agoYou could try the tool my group builds, Arvados (https://arvados.org https://arvados.org). We use Common Workflow Language (CWL) as the workflow language. Arvados works great for very large computations and data management at petabyte scale. It really shines in a production environment where data provenance is key. Intelligent handling of failures (which are inevitable at scale) is a key part of the Arvados design.