2 ms·
> far preferable Well, that depends. Shell scripts can be commented, and they can be built progressively and interactively by building a pipeline. That's a g
by Symbiote 10y ago
> far preferable
Well, that depends. Shell scripts can be commented, and they can be built progressively and interactively by building a pipeline. That's a great choice for a one-off task, and in my experience much faster than many other approaches.
It also works well for tasks close to the system. For example, our users are able to download large archives of data, and we keep over 99% of such downloads indefinitely. We delete downloads > 100GB after they're more than 6 months old.
With a shell script run by cron that's achieved with find + rm + curl|jq (to tell the API the download is deleted).
- minimaxir 10y agoTo clarify, by data processing I mean extract-transform-load, where the outputs of the extract/transform phase(s) may not be immediately obvious. Bash as an simple automation tool for simple tasks is fine.