3 ms·

Can't you just ask one of the machines in the databricks/spark cluster to run those shell commands? Or is that more of a kubernetes thing?

• mr_toad 5y ago
  You can use Linux shell commands in Map Reduce jobs using Hadoop streaming.
• westurner 5y ago
  That's pretty much how Dask works; though Dask doesn't throw away data types for newline-delimited strings between each failed partitioned IPC pipeline process, and then the new hire didn't appropriately trap errors in their shell script, so the log messages are chronologically non-sequential and text-only. https://github.com/dask/dask-labextension https://github.com/dask/dask-labextension : > This package provides a JupyterLab extension to manage Dask clusters, as well as embed Dask's dashboard plots directly into JupyterLab panes. Something like ml-hub allows MLops teams to create resource-quota'd containers with k8s and IAM, though even signed code can DoS an unauditable system with no logs of which processes ran which signed archive of which code at what time, with bash and ssh. https://github.com/ml-tooling/ml-hub https://github.com/ml-tooling/ml-hub