5 ms·

>> The crazy think is we still are extremely bad at fitting things together - still the best way of fitting things together is the unix pipe Is that sad or pur

• laumars 14y ago
  I struggled with that point of his. From a coding speed perspective, he's right. But from an efficiency perspective, even some of the slowest interpreted languages will run circles around a shell script. However it's still an interesting and thought provoking point. It makes you wonder about other ways of plugging different objects / modules together.
  • bo1024 14y ago
    He doesn't mean literally the "|" in bash, he means the general concept.
    • laumars 14y ago
      In a sense, it's the same thing. He's talking about the concept of fitting things together and refers to Unix pipes (in fact the example he gave was literally a string of piped POSIX commands ready for dumping into $SHELL). My point was that this concept doesn't scale well. As a productivity tool, pipes are invaluable. But you wouldn't want to write a performance critical routine using them. At the end of the day (and as Joe said himself), it's about picking the right tool for the job.
      • disgruntledphd2 14y ago
        What's the difference between pipes and chaining functions? I can't really see much difference between grep | sed |awk somestuff.txt and (awk(sed(grep(somestuff.txt))). Or are you suggesting that function composition is not the right approach? In which case, I disagree, but I would like some more insight into your thought process on this.
        • anon1385 14y ago
          I can't speak for laumars, but I would suggest there is a difference. Functions within languages generally work at a higher level of abstraction, even in C. If you wanted a grep like function for filtering an array of strings in your programming language of choice, would you define it to take a single string and predicate arguments, that is then split up on an arbitrary character, filtered, then joined back together again to return a single string? Or would you use a general purpose filter() function that takes an array and a predicate (function pointer/lambda/block/whatever) and returns a new array?
        • laumars 14y ago
          My issue largely about the scalability of pipes. They're ok for basic tasks, but not great for complicated routines: 1) You've only got one send and one return. (well, arguably more if you include stderr and exit codes, but they have their own limitations in addition to the aforementioned) 2) They're too insular from each other; different files dotted about that needs to be loaded into the memory then executed. It adds quite a massive overhead. Granted this isn't an issue for the kind of jobs you'd write shell scripts for in the first place, but it does severely hamper this kind of model in terms of scalability. But then we're back to the age old issue of performance vs convenience. I will concede that the second point is more an issue about implementation rather than concept though. But if you were to write a lower level implementation of Unix pipes, I don't think it would pretty much end up with Perl functions (which, to be fair, you suggested yourself). I say Perl specifically because you can create a function without specifying what values to accept (see example below), which is akin to the 'dumb' strings read in from STDIN. sub PerlFunc() { foreach (@_) { $i++; print "Parameter $i == $_\n"; } } So I'm not really saying that pipes isn't the "right approach". Just that it isn't always the best approach. But for things like system administration, pipes are invaluable. Not just because the tools are already written (ie the wealth of command line applications), but also because it's quick to express yet highly readable. I'm not really sure if that answers your question though (or even if I've said anything you don't already know).
          • kd0amg 14y ago
            I'm not really sure if that answers your question though I think GP was confused about the difference between the concept of piping and the concept of function composition. As noted by GGGP above, the concept of piping/composition is not only Unix pipes, but you keep arguing about that particular case of it instead of the general concept.
      • dragonwriter 14y ago
        Note that that example isn't given as an example of what would be ideal, but as a sign of the sad state of tools to fit things together that it is the best available.
  • fusiongyro 14y ago
    "Today there is an unhealthy concentration on language and efficiency and NOT on how things fit together and protocols." -- Joe, two paragraphs below the point we're discussing.
    • laumars 14y ago
      I think you're being a little unfair there because he's talking about an "unhealthy concentration". Using a lower level language instead of a shell script for high performance scripts isn't unhealthy as it's not a case of just saving a negligible number of clock cycles. The difference I'm talking about like travelling to the moon and back just to buy a pint of milk. Case in point: no sane person would rewrite Apache in Bash. But equally I wouldn't rewrite any of my sys admin shell scripts in C. There's a balance that needs to be struck. Which I did also hint at in my previous post (the one you dismissed outright when arguing about 'unhealthy concentrations'). And I think what Joe actually meant by that quote was the same thing; that many developers are bad at judging that balance.
      • fusiongyro 14y ago
        I think you're reading a bit much into my intent from a comment that is really just a quote from the article.
        • laumars 14y ago
          It seems a little odd to specifically single out my post about efficiency with a quote about efficiency if your intent wasn't to address my point. So if I am reading too much into your post, then what was the intent?
          • fusiongyro 14y ago
            I'm sorry; I wasn't trying to bludgeon you with Joe's words. Looking back it is clear that I did and I should have added some exposition of my own. However, my goal was to see more about where you draw the line, and I think you did answer that, so thanks.
  • anon1385 14y ago
    Is efficiency the biggest problem? It seems to me the bigger problem is that streams of raw bytes over pipes is the wrong level of abstraction for most tasks; at least we seem to have decided that is the case when we are writing code within a program, so I don't see why it shouldn't extend to interprocess programming. I don't see people recommending only defining functions that accept single dimensional arrays of bytes (rather than strings and trees and hashmaps and multidimensional arrays and so on), even when programming in C. You end up with every tool containing code to parse a stream of bytes into the appropriate data structures, and then probably also write them out to a stream of bytes too. The user has responsibility for a lot of data munging too. It is error prone and repetitive, at least when dealing with anything other than very simple text files containing lines of strings where you know which encoding has been used. (Re)using libraries that read and write more structured data is likely to get you flamed[1]. The history of Unix contains plenty of security problems due to people not correctly accounting for nulls or control characters or spaces etc when chaining together commands (although this is more of a problem with shells than pipes themselves). The output is often not friendly to human eyes by default (I'm thinking of things like localised date and time formats, number formats and so on) since it has to be suitable for passing to another program, unless you use more options or another tool to reformat it. I would compare it to working with raw memory in C. It's the lowest common denominator, you can do anything, but it's also trivial to screw up and there is a high cognitive overhead. Perhaps it's just too hard to introduce anything higher level at this point. Perhaps my opinion would change if I spent years (deeply) learning unix tools, but maybe that would just be because I had invested so much time learning a complicated system.
    • laumars 14y ago
      Unix pipes are effectively the same thing as using text streams in C. Granted not quite the same thing, but raw bits are often used in lower level languages. Take Windows Win32 APIs, there's a lot of instances where styles are defined by adding constants with values being exponentials of 2. Thus creating a binary array of boolean states. Another example I used to run into was Windows' controller (joystick et al) API (again Win32). Each bit would represent a different controller button and the 'on' state was if the button was depressed. But as the value was returned as an unsigned long int (if memory serves), it was up to the developer to write their own parser to convert what would otherwise been a random number into a meaningful array of bits. (or at least I did - there is a chance I overlooked another function as this was before I made the switch to DirectX6 - so many years ago!)
• jimbokun 14y ago
  Isn't the modern equivalent JSON over a REST API?
  • astine 14y ago
    That's only for web applications though. I'd like it if Unix utilities optionally output and accepted JSON or a similar syntax. It'd make things so much easier. http://theatticlight.net/posts/Why-cant-we-do-pipes-smarter/ http://theatticlight.net/posts/Why-cant-we-do-pipes-smarter/