4 ms·

It often isn’t about the execution time, if it’s a daily job and takes 26 minutes who cares? Using something like databricks means it is easy to schedule and m

It often isn’t about the execution time, if it’s a daily job and takes 26 minutes who cares? Using something like databricks means it is easy to schedule and manage jobs, easy to write jobs that work in good enough time, easy to troubleshoot when things go wrong. It comes with a well documented security mode and a support contract when needed. Developers can be onboarded quickly and work code reviewed and managed. Logging and diagnostics are available and you can report on metrics easily. That isn’t true with custom data pipelines written in shell scripts. The value isn’t in the pure execution time, it is in everything around it.

jbverschoor 5y ago
Processes that take ages are never "easy to write" nor "easy to troubleshoot". This turns into shitty code, because you won't get any sane person to spend weeks instead of days on that. It's bad for morale. It's the same narrative people use for many complexity or slow things
wodenokoto 5y ago
Can't you just ask one of the machines in the databricks/spark cluster to run those shell commands? Or is that more of a kubernetes thing?
- mr_toad 5y ago
  You can use Linux shell commands in Map Reduce jobs using Hadoop streaming.
- westurner 5y ago
  That's pretty much how Dask works; though Dask doesn't throw away data types for newline-delimited strings between each failed partitioned IPC pipeline process, and then the new hire didn't appropriately trap errors in their shell script, so the log messages are chronologically non-sequential and text-only. https://github.com/dask/dask-labextension https://github.com/dask/dask-labextension : > This package provides a JupyterLab extension to manage Dask clusters, as well as embed Dask's dashboard plots directly into JupyterLab panes. Something like ml-hub allows MLops teams to create resource-quota'd containers with k8s and IAM, though even signed code can DoS an unauditable system with no logs of which processes ran which signed archive of which code at what time, with bash and ssh. https://github.com/ml-tooling/ml-hub https://github.com/ml-tooling/ml-hub
_pvxk 5y ago
> The value isn’t in the pure execution time, it is in everything around it. Especially for those of us who get paid for developing, designing, supporting, architecturing, meetinging, catering, conferencing and maintaining everything around it :-)
usrbinbash 5y ago
> That isn’t true with custom data pipelines written in shell scripts. Why not? Cron can schedule jobs, the documentation for standard shell tools is among the best, almost every developer can handle bash scripts, Logging can be done via syslog.
- skocznymroczny 5y ago
  This is like the classic Dropbox hackernews comment: > For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software. Sure, you can stitch several services together and it will work for your needs, but for most users there is a benefit to a centrally managed and complete solution.
  - usrbinbash 5y ago
    > but for most users Define "most users" Most users who have to tackle actual big-data problems, meaning analysing things on the order of several TB or more? Sure, they will absolutely benefit. But there isn't just big data. There is also little data, where what is analysed is on the order of a few GiB or less, and everything in between. I am not saying "use shell for everything!" I am saying "the right tool for the right job". A 15t excavator is probably not a good choice if I want to plant a small tree in my backyard, and a gardening shovel will probably not serve me well when I wanna start building a scyscraper.
BasDirks 5y ago
I don't see why this is downvoted. Seems like a solid argument, even though I don't agree.

jbverschoor 5y ago

Processes that take ages are never "easy to write" nor "easy to troubleshoot". This turns into shitty code, because you won't get any sane person to spend weeks instead of days on that. It's bad for morale. It's the same narrative people use for many complexity or slow things

wodenokoto 5y ago

Can't you just ask one of the machines in the databricks/spark cluster to run those shell commands? Or is that more of a kubernetes thing?

mr_toad 5y ago
You can use Linux shell commands in Map Reduce jobs using Hadoop streaming.
westurner 5y ago
That's pretty much how Dask works; though Dask doesn't throw away data types for newline-delimited strings between each failed partitioned IPC pipeline process, and then the new hire didn't appropriately trap errors in their shell script, so the log messages are chronologically non-sequential and text-only. https://github.com/dask/dask-labextension https://github.com/dask/dask-labextension : > This package provides a JupyterLab extension to manage Dask clusters, as well as embed Dask's dashboard plots directly into JupyterLab panes. Something like ml-hub allows MLops teams to create resource-quota'd containers with k8s and IAM, though even signed code can DoS an unauditable system with no logs of which processes ran which signed archive of which code at what time, with bash and ssh. https://github.com/ml-tooling/ml-hub https://github.com/ml-tooling/ml-hub

_pvxk 5y ago

> The value isn’t in the pure execution time, it is in everything around it. Especially for those of us who get paid for developing, designing, supporting, architecturing, meetinging, catering, conferencing and maintaining everything around it :-)

usrbinbash 5y ago

> That isn’t true with custom data pipelines written in shell scripts. Why not? Cron can schedule jobs, the documentation for standard shell tools is among the best, almost every developer can handle bash scripts, Logging can be done via syslog.

skocznymroczny 5y ago
This is like the classic Dropbox hackernews comment: > For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software. Sure, you can stitch several services together and it will work for your needs, but for most users there is a benefit to a centrally managed and complete solution.
- usrbinbash 5y ago
  > but for most users Define "most users" Most users who have to tackle actual big-data problems, meaning analysing things on the order of several TB or more? Sure, they will absolutely benefit. But there isn't just big data. There is also little data, where what is analysed is on the order of a few GiB or less, and everything in between. I am not saying "use shell for everything!" I am saying "the right tool for the right job". A 15t excavator is probably not a good choice if I want to plant a small tree in my backyard, and a gardening shovel will probably not serve me well when I wanna start building a scyscraper.

BasDirks 5y ago

I don't see why this is downvoted. Seems like a solid argument, even though I don't agree.