4 ms·
> That isn’t true with custom data pipelines written in shell scripts. Why not? Cron can schedule jobs, the documentation for standard shell tools is among the
by usrbinbash 5y ago
> That isn’t true with custom data pipelines written in shell scripts.
Why not? Cron can schedule jobs, the documentation for standard shell tools is among the best, almost every developer can handle bash scripts, Logging can be done via syslog.
- skocznymroczny 5y agoThis is like the classic Dropbox hackernews comment: > For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software. Sure, you can stitch several services together and it will work for your needs, but for most users there is a benefit to a centrally managed and complete solution.
- usrbinbash 5y ago> but for most users Define "most users" Most users who have to tackle actual big-data problems, meaning analysing things on the order of several TB or more? Sure, they will absolutely benefit. But there isn't just big data. There is also little data, where what is analysed is on the order of a few GiB or less, and everything in between. I am not saying "use shell for everything!" I am saying "the right tool for the right job". A 15t excavator is probably not a good choice if I want to plant a small tree in my backyard, and a gardening shovel will probably not serve me well when I wanna start building a scyscraper.