3 ms·
The problem is that the pipeline model is extremely fragile and breaks in unexpected ways in unexpected places when hit with the real world. The need to handle
by tsimionescu 6y ago
The problem is that the pipeline model is extremely fragile and breaks in unexpected ways in unexpected places when hit with the real world.
The need to handle spaces and quotes can take you from a 20 character pipeline to a 10 line script, or a C program. That is not a good model whichever way you look at it.
- diegocg 6y agoThat's not the fault of pipes, that's the fault of shells like bash. There are shells that deal with spaces and quotes perfectly fine.
- necrotic_comp 6y agoI've been scripting stuff on the pipeline for over a decade and haven't really run into this much. You can define a field separator in the command line with the environment variable IFS - i.e. 'IFS=$(echo -en "\n\b");' for newlines - which takes care of the basic cases like spaces in filenames/directory names when doing a for loop, and if I have other highly structured data that is heavily quoted or has some other sort of structure to it, then I either normalize it in some fashion or, as you suggest, write a perl script. I haven't found it too much of a burden, even when dealing with exceptionally large files.
- mbreese 6y agoPipelines are mainly for short-lived, one-off quick scripts. But they are also really useful for when you control the input data. For example, if you need a summary report based on the results of a sql query. If you control the inputs and you need to support quotes, spaces, non-white space delimiters, etc... in shell script, then that’s on you. If you don’t control the inputs, then shell scripts are generally a poor match. For example, if you need summary reports from a client, but they sometimes provide the table in xlxs or csv format — shell might not be a good idea. Might be controversial, but I think you can tell who works with shell pipes the most by looking at who uses CSV vs tab-delimited text files. Tabs can still be a pain if you have spaces in data. But if you mix shell scripts with CSV, you’re just asking for trouble.
- nerdponx 6y agoKind of / not really. Also Zsh solves 99% of my "pipelines are annoying" and "shell scripts are annoying" problems. Even on systems where I can't set my default shell to Zsh, I either use it anyway inside Tmux, or I just use it for scripting. I suppose I use Zsh the way most people use Perl.
- xkucf03 6y agoI agree that spaces, quotes, encodings, line-ends etc. are problem, but this is not problem of „the pipeline model“ – this is problem of ambiguously-structured data. See https://relational-pipes.globalcode.info/v_0/classic-example.xhtml https://relational-pipes.globalcode.info/v_0/classic-example...