26 ms·

Unix piping is basically functional programming. If you ever wondered why some people are obsessed with functional programming this is the reason why: Functio

Unix piping is basically functional programming. If you ever wondered why some people are obsessed with functional programming this is the reason why: Functional programming forces every primitive in your program to be a Lego Block. A lot of functional programmers don't see the big picture. They see a sort of elegance with the functional style, they like the immutability but they can't explain the practical significance to the uninitiated. Functional Programming is the answer to the question that has plagued me as a programmer for years. How do I organize my program in such a way that it becomes endlessly re-useable from a practical standpoint? Functional programming transforms organ transplantation into lego building blocks. The "lego" is the "function" and "connecting two lego blocks" is "function composition". In short, another name for "Point free style" programming is "Lego building block style" programming.

EGreg 6y ago
Want to build web apps from reusable blocks? Reason at a higher level about chatrooms, roles and permissions, credits? That was the thinking behind our open source project: https://qbix.com/platform https://qbix.com/platform Reusability on the web. Here is where we are going: https://qbix.com/QBUX/whitepaper.html#Distributed-Operating-System https://qbix.com/QBUX/whitepaper.html#Distributed-Operating-...
- adamsea 6y ago
  Is there a sample project somewhere on Github? Would be helpful to see a concrete example of use to get a better idea of it.
xvedejas 6y ago
You can't just compose two functions because they're both written in a functional programming language. The programmer has to have the foresight to make their types compatible. I think the novelty of the Unix pipe for interoperability is that they (typically) work on one agreed-upon kind of data: human readable, white-space separated. So a lot of tools "just work" with each other. There's no reason you can't do this with functional programming, but obviously you can do it with non-functional programming too, and you could certainly fail to do this with functional programming.
- crimsonalucard5 6y ago
  >You can't just compose two functions because they're both written in a functional programming language. Obviously not, the types have to be compatible just like how a lego piece must be compatible. >The programmer has to have the foresight to make their types compatible. Just like a lego builder needs to have the foresight to see whether two lego components are compatible. The analogy still fits. > I think the novelty of the Unix pipe for interoperability is that they (typically) work on one agreed-upon kind of data: human readable, white-space separated. So a lot of tools "just work" with each other. The agreed upon data is just a type. Unix pipes compose functions that take in strings as input and deliver strings as output. The spaces is just an oversight, the whole thing would have been more clean if the type was designed to represent multiple pieces of data like a tuple of words rather than a single string with words divided by spaces. >you could certainly fail to do this with functional programming. Just like how two lego pieces can't compose if they're not compatible pieces facing the right direction. Sockets and plugs must be compatible. A functional program still fits the analogy of legos perfectly. Take a look at lego pieces here: https://brickarchitect.com/2019/2019-most-common-lego-parts/ https://brickarchitect.com/2019/2019-most-common-lego-parts/ There exists lego parts that can never compose with certain other parts. Just like functions and function composition. What you're not seeing is how other styles of programming such as OOP or procedural programming fail to fit the analogy and literally become like organ transplantation. Let's take a look at object composition. How does that work in the context of legos? It's like a lego block with a mutating hole in it and once you can put different things in that hole and that changes the overall lego block. Also with OOP and procedural programming you get lego blocks that can mutate. Lego blocks can transform themselves and other lego blocks. They form an interconnected network of entities that are constantly mutating. To fit the analogy of a lego block you need unchanging blocks. The entire field of Procedural programming and OOP is basically the art of building systems with mutating primitives that constantly change. Imagine constructing buildings using bricks that change shape. Now imagine refactoring code that does this.... Organ transplantation. Also you really just need to try it. Doing the type of grafting and refactoring that's an intrinsic part of programming literally becomes lego-like once you program using the functional style. To really see the big picture I suggest you exclusively try the point free style using Haskell. That being said. There are still massive tradeoffs to doing functional programming. But if your goal is to program like you're using lego blocks, functional programming is the path.
czbond 6y ago
You really just turned functional programming around for me. I learned CompSci object oriented (C++) but have always loved how easy data analysis was in unix output. Cheers for making me want to give it another look!
- crimsonalucard5 6y ago
  Cool! Glad I was able to convince you. Just note that a lot of people find the functional style kind of unnecessarily mind bending. Like the functional style primarily uses recursion instead of loops for iteration which is harder to reason about (and less performant). I actually agree with them, but I feel that these people got lost in the details and missed the big picture about the benefits of functional programming from the perspective of modules, design and program organization. I recommend you try building something complex with the point free style using Haskell. It will be a bit mind bending at first but if you get it then you'll see how FP is basically the same thing as using legos to build data pipelines... exactly the style used with unix pipes.
BiteCode_dev 6y ago
> Unix piping is basically functional programming Except you litterally use side effects to communicate. Not really FP, that part.
- schoen 6y ago
  How so? While programs themselves could have side effects, the pipe -- if its elements are deterministic which is indeed not true of everything -- is just a series of composed transformations of text. In a pipeline like seq 100 | number | grep '[a-z]' | sort | tr -d '.' | tr a-z A-Z every component is a pure function (byte strings -> byte strings) and the pipeline's results are totally deterministic [if you don't change the locale or collation order with environment variables]. No component of the pipeline makes any change to the filesystem¹. ¹ although both stat(2) and inotify(7) mechanisms could allow other processes to detect the interactions with the filesystem that are caused by running the pipeline, among other process-related mechanisms that system as a whole could use to detect how many times the pipeline has been run, so Unix itself is definitely not side-effect-free for almost any operation
  - BiteCode_dev 6y ago
    > [if you don't change the locale or collation order with environment variables] Or any other state change. Because it's not stateless. Hence not FP.
    - crimsonalucard5 6y ago
      Unless you use some stateful program or you're writing to a file typically a unix expression that uses purely reads and pipes is deterministic meaning you run the same expression twice you get the same result. If you want to get pedantic fine, but the majority of use cases is stateless.
      - schoen 6y ago
        It's kind of an interesting pattern to think of having the pipeline as a whole be purely deterministic, but its initial input be something from the environment (kind of like the Haskell IO monad, right?). Probably most complex shell pipelines follow this pattern in practice, but the first counterexample I thought of was shuf, or sort -R. Also, many pipelines that include any kind of interpolation or variable substitution also don't follow it. A different insight about reasoning about pipelines might be that the byte string (or ASCII string) type is too weak to catch most kinds of errors, and it's not uncommon for one pipeline component to not, in fact, be a total function with respect to the actual data type that you're trying to capture (I don't know the right terminology for this). This most famously happens when you do regular expression substitutions but your regular expression doesn't actually match the full grammar that you're looking for. Then the pipeline can be incorrect as a whole for some inputs, but earlier and later stages don't notice. It can also happen anywhere that different tools have a different implicit understanding of the relevant grammar or structure. That connects up with all sorts of other ideas which are actually about type safety and parsing more than FP. For example, PowerShell has tried to take the pipeline concept in a different direction, to the consternation of us Unix purists. Its use of typed objects in this context makes more explicit what the contract between programs in the pipeline is supposed to be. There is also a LANGSEC connection in terms of the risks of informal or underspecified parsers and grammars. I know I've personally written lots of Unix pipelines that were correct for all the inputs that I personally threw at them, but definitely not correct for every possible input. I like to use ! in vim frequently to shell out to a command line to perform a text editing task, and vim has an associated undo and redo which means that sometimes I'm trying several variants until I find the one that successfully appears to do the edit that I intended. Sometimes I do this at an almost preconscious level, which is really strange in terms of thinking about the concept of the correctness of a program (in this case, where the program is literally only going to be used once, with the programmer looking over its shoulder).
hellofunk 6y ago
I think this is a rather simplified and naïve analysis. Getting functional programs and functional APIs to compose well with each other is just as much a challenge as in other language paradigms. Just because the logic is organized as functions doesn’t magically make things fit together. Your APIs need to speak in a consistent way as well, and the arrangement of your data needs to be the same or easily convertible between your “Lego pieces“. Having spent many years of my career writing functional code all day, it is just as easy to make a mess of things in functional programs as it is an object oriented programs. I do not believe either is inherently better at creating the “Lego“–style.
- willtim 6y ago
  It has to be pure functional programming to get the full benefits of compositionality. Most mainstream "functional programming" is not necessarily pure and side-effects are not controlled/tracked. Analogous to how "mostly secure" is not secure, "mostly functional" does not get the full benefits. Pure functional programming is complex, one has to compose effectful functions differently to pure functions. But the pieces do really fit like Lego bricks, especially when the same mathematical abstractions are used consistently. The Haskell community has been extremely effective at creating such consistency, by promoting various abstractions using category theory as a guide. The object-oriented community is not so different in this regard with their promotion of "patterns". I've been writing Haskell professionally for 8 years. The problems with Haskell are the tooling, language stability, the learning curve and the difficulty of reasoning about performance, especially space usage. But composition and re-use works.
  - hellofunk 6y ago
    I'm sorry but this is a lot of shallow hyperbole. It doesn't matter if your functions are pure if they return a unique data type for your API or library that is not understood easily by the caller. So the solution is to make that data easily convertible or generic.. and guess what? That's no different than doing the same in an OO language. In the end, lego pieces are ultimately about the data itself, not how it is manipulated. The key ingredient in making software reusable and portable is the talent and experience of the engineer, regardless of language.
mpweiher 6y ago
> Unix piping is basically functional programming. Only in the same sense that all computing is Turing Machines or NAND gates. This is a very common misunderstanding, but there is a reason that FP, which is transformational, had to adopt dataflow in order to sort-of handle reactive systems. Functions run to completion and return their result. Filters tend to run concurrently and, importantly, do not return their results, they pass them on to the next filter in the pipeline. It is possible to compose any two filters. It is not possible to compose any two functions, not even close, the default for functions is to not compose (arity, parameter types, return type,...).
- crimsonalucard5 6y ago
  >Filters tend to run concurrently and, importantly, do not return their results, they pass them on to the next filter in the pipeline. Filters are functions. They are one and the same. For unix it's a function that takes in a string and outputs a string. The unix world reduces everything into a singular type. You will note that unix "filters" cannot compose with functions of "other" types either. If unix included types like arrays or ints in stdout you would have the same typing problems as you have with functions. In the unix world all your types are strings so things seem simpler, but in reality your little unix programs need to deserialize the strings into proper types in order to do anything meaningful. You could achieve the same thing if you used functions that returned strings all the time then did internal deserialization but that's not really a solution. I do get your point about arity and parameter types. You're referring to the fact that not all functions can compose UNLESS they have compatible types and an arity of 1 in the parameters and an arity of 1 in the return value (Golang for example can have arity > 1 in the return value). However this arity issue doesn't really exist. A function with Arity of two is isomorphic to a function of arity 1 that takes in a 2-tuple as input. f(a, b) -> c f'(Tuple[a, b]) -> c g(d) -> Tuple[a, b] Both f' and f are equivalent in theory. Plus you can compose g with f'. The only compatibility problems with composition among functions in the end is just types (not arity) but this problem still exists even in the unix world... it's just hidden from you because you don't actually watch the programs perform deserialization and serialization.
  - 6y ago
c3534l 6y ago
The goal of OOP is also to be modular and composable. I think the thing that is a good design choice is modular and composible. It's what the industrial revolution / assembly line was based on. It's what vim keybindings are based on. Heck, it's what programming languages themselves are based on. Here are some simple tools that do easily understandable things together, now put something together with it. Lego bricks are fun and useful and it's not the domain of any one area of CS.
- crimsonalucard5 6y ago
  >The goal of OOP is also to be modular and composable. Well it depends on the type of composition you're talking about. Anything when mashed together hard enough can compose.
heavyset_go 6y ago
One can say similar things about interfaces in object-oriented programming.
- AnimalMuppet 6y ago
  Not really. OOP interfaces are usually much more intricate than FP interfaces. You might say that they are less like a lego interface and more like electronic connectors. There are several different electronic connectors types, and they are designed so that you physically can't plug the wrong things together. OOP is like that - a more complicated interface that makes it impossible to connect anything-to-anything the way legos can. On the other hand, if you're trying to build something large (a house, say - a real house, not a toy one), you don't want the lego interface. Sure, you can plug anything to anything, but not all of those connections make sense. Also, for building something that large, legos are too small a building block to be convenient. You want some larger things - beams and sheets of wood and particleboard, pipes, air ducts. You don't want to have to build all of those out of tiny blocks. In the same way, I wonder if building larger applications out of FP is going to be similarly tedious. I have never done it, so I will admit that I don't know.

EGreg 6y ago

Want to build web apps from reusable blocks? Reason at a higher level about chatrooms, roles and permissions, credits? That was the thinking behind our open source project: https://qbix.com/platform https://qbix.com/platform Reusability on the web. Here is where we are going: https://qbix.com/QBUX/whitepaper.html#Distributed-Operating-System https://qbix.com/QBUX/whitepaper.html#Distributed-Operating-...

adamsea 6y ago
Is there a sample project somewhere on Github? Would be helpful to see a concrete example of use to get a better idea of it.

xvedejas 6y ago

You can't just compose two functions because they're both written in a functional programming language. The programmer has to have the foresight to make their types compatible. I think the novelty of the Unix pipe for interoperability is that they (typically) work on one agreed-upon kind of data: human readable, white-space separated. So a lot of tools "just work" with each other. There's no reason you can't do this with functional programming, but obviously you can do it with non-functional programming too, and you could certainly fail to do this with functional programming.

crimsonalucard5 6y ago
>You can't just compose two functions because they're both written in a functional programming language. Obviously not, the types have to be compatible just like how a lego piece must be compatible. >The programmer has to have the foresight to make their types compatible. Just like a lego builder needs to have the foresight to see whether two lego components are compatible. The analogy still fits. > I think the novelty of the Unix pipe for interoperability is that they (typically) work on one agreed-upon kind of data: human readable, white-space separated. So a lot of tools "just work" with each other. The agreed upon data is just a type. Unix pipes compose functions that take in strings as input and deliver strings as output. The spaces is just an oversight, the whole thing would have been more clean if the type was designed to represent multiple pieces of data like a tuple of words rather than a single string with words divided by spaces. >you could certainly fail to do this with functional programming. Just like how two lego pieces can't compose if they're not compatible pieces facing the right direction. Sockets and plugs must be compatible. A functional program still fits the analogy of legos perfectly. Take a look at lego pieces here: https://brickarchitect.com/2019/2019-most-common-lego-parts/ https://brickarchitect.com/2019/2019-most-common-lego-parts/ There exists lego parts that can never compose with certain other parts. Just like functions and function composition. What you're not seeing is how other styles of programming such as OOP or procedural programming fail to fit the analogy and literally become like organ transplantation. Let's take a look at object composition. How does that work in the context of legos? It's like a lego block with a mutating hole in it and once you can put different things in that hole and that changes the overall lego block. Also with OOP and procedural programming you get lego blocks that can mutate. Lego blocks can transform themselves and other lego blocks. They form an interconnected network of entities that are constantly mutating. To fit the analogy of a lego block you need unchanging blocks. The entire field of Procedural programming and OOP is basically the art of building systems with mutating primitives that constantly change. Imagine constructing buildings using bricks that change shape. Now imagine refactoring code that does this.... Organ transplantation. Also you really just need to try it. Doing the type of grafting and refactoring that's an intrinsic part of programming literally becomes lego-like once you program using the functional style. To really see the big picture I suggest you exclusively try the point free style using Haskell. That being said. There are still massive tradeoffs to doing functional programming. But if your goal is to program like you're using lego blocks, functional programming is the path.

czbond 6y ago

You really just turned functional programming around for me. I learned CompSci object oriented (C++) but have always loved how easy data analysis was in unix output. Cheers for making me want to give it another look!

crimsonalucard5 6y ago
Cool! Glad I was able to convince you. Just note that a lot of people find the functional style kind of unnecessarily mind bending. Like the functional style primarily uses recursion instead of loops for iteration which is harder to reason about (and less performant). I actually agree with them, but I feel that these people got lost in the details and missed the big picture about the benefits of functional programming from the perspective of modules, design and program organization. I recommend you try building something complex with the point free style using Haskell. It will be a bit mind bending at first but if you get it then you'll see how FP is basically the same thing as using legos to build data pipelines... exactly the style used with unix pipes.

BiteCode_dev 6y ago

> Unix piping is basically functional programming Except you litterally use side effects to communicate. Not really FP, that part.

schoen 6y ago
How so? While programs themselves could have side effects, the pipe -- if its elements are deterministic which is indeed not true of everything -- is just a series of composed transformations of text. In a pipeline like seq 100 | number | grep '[a-z]' | sort | tr -d '.' | tr a-z A-Z every component is a pure function (byte strings -> byte strings) and the pipeline's results are totally deterministic [if you don't change the locale or collation order with environment variables]. No component of the pipeline makes any change to the filesystem¹. ¹ although both stat(2) and inotify(7) mechanisms could allow other processes to detect the interactions with the filesystem that are caused by running the pipeline, among other process-related mechanisms that system as a whole could use to detect how many times the pipeline has been run, so Unix itself is definitely not side-effect-free for almost any operation
- BiteCode_dev 6y ago
  > [if you don't change the locale or collation order with environment variables] Or any other state change. Because it's not stateless. Hence not FP.
  - crimsonalucard5 6y ago
    Unless you use some stateful program or you're writing to a file typically a unix expression that uses purely reads and pipes is deterministic meaning you run the same expression twice you get the same result. If you want to get pedantic fine, but the majority of use cases is stateless.
    - schoen 6y ago
      It's kind of an interesting pattern to think of having the pipeline as a whole be purely deterministic, but its initial input be something from the environment (kind of like the Haskell IO monad, right?). Probably most complex shell pipelines follow this pattern in practice, but the first counterexample I thought of was shuf, or sort -R. Also, many pipelines that include any kind of interpolation or variable substitution also don't follow it. A different insight about reasoning about pipelines might be that the byte string (or ASCII string) type is too weak to catch most kinds of errors, and it's not uncommon for one pipeline component to not, in fact, be a total function with respect to the actual data type that you're trying to capture (I don't know the right terminology for this). This most famously happens when you do regular expression substitutions but your regular expression doesn't actually match the full grammar that you're looking for. Then the pipeline can be incorrect as a whole for some inputs, but earlier and later stages don't notice. It can also happen anywhere that different tools have a different implicit understanding of the relevant grammar or structure. That connects up with all sorts of other ideas which are actually about type safety and parsing more than FP. For example, PowerShell has tried to take the pipeline concept in a different direction, to the consternation of us Unix purists. Its use of typed objects in this context makes more explicit what the contract between programs in the pipeline is supposed to be. There is also a LANGSEC connection in terms of the risks of informal or underspecified parsers and grammars. I know I've personally written lots of Unix pipelines that were correct for all the inputs that I personally threw at them, but definitely not correct for every possible input. I like to use ! in vim frequently to shell out to a command line to perform a text editing task, and vim has an associated undo and redo which means that sometimes I'm trying several variants until I find the one that successfully appears to do the edit that I intended. Sometimes I do this at an almost preconscious level, which is really strange in terms of thinking about the concept of the correctness of a program (in this case, where the program is literally only going to be used once, with the programmer looking over its shoulder).

hellofunk 6y ago

I think this is a rather simplified and naïve analysis. Getting functional programs and functional APIs to compose well with each other is just as much a challenge as in other language paradigms. Just because the logic is organized as functions doesn’t magically make things fit together. Your APIs need to speak in a consistent way as well, and the arrangement of your data needs to be the same or easily convertible between your “Lego pieces“. Having spent many years of my career writing functional code all day, it is just as easy to make a mess of things in functional programs as it is an object oriented programs. I do not believe either is inherently better at creating the “Lego“–style.

willtim 6y ago
It has to be pure functional programming to get the full benefits of compositionality. Most mainstream "functional programming" is not necessarily pure and side-effects are not controlled/tracked. Analogous to how "mostly secure" is not secure, "mostly functional" does not get the full benefits. Pure functional programming is complex, one has to compose effectful functions differently to pure functions. But the pieces do really fit like Lego bricks, especially when the same mathematical abstractions are used consistently. The Haskell community has been extremely effective at creating such consistency, by promoting various abstractions using category theory as a guide. The object-oriented community is not so different in this regard with their promotion of "patterns". I've been writing Haskell professionally for 8 years. The problems with Haskell are the tooling, language stability, the learning curve and the difficulty of reasoning about performance, especially space usage. But composition and re-use works.
- hellofunk 6y ago
  I'm sorry but this is a lot of shallow hyperbole. It doesn't matter if your functions are pure if they return a unique data type for your API or library that is not understood easily by the caller. So the solution is to make that data easily convertible or generic.. and guess what? That's no different than doing the same in an OO language. In the end, lego pieces are ultimately about the data itself, not how it is manipulated. The key ingredient in making software reusable and portable is the talent and experience of the engineer, regardless of language.

mpweiher 6y ago

> Unix piping is basically functional programming. Only in the same sense that all computing is Turing Machines or NAND gates. This is a very common misunderstanding, but there is a reason that FP, which is transformational, had to adopt dataflow in order to sort-of handle reactive systems. Functions run to completion and return their result. Filters tend to run concurrently and, importantly, do not return their results, they pass them on to the next filter in the pipeline. It is possible to compose any two filters. It is not possible to compose any two functions, not even close, the default for functions is to not compose (arity, parameter types, return type,...).

crimsonalucard5 6y ago
>Filters tend to run concurrently and, importantly, do not return their results, they pass them on to the next filter in the pipeline. Filters are functions. They are one and the same. For unix it's a function that takes in a string and outputs a string. The unix world reduces everything into a singular type. You will note that unix "filters" cannot compose with functions of "other" types either. If unix included types like arrays or ints in stdout you would have the same typing problems as you have with functions. In the unix world all your types are strings so things seem simpler, but in reality your little unix programs need to deserialize the strings into proper types in order to do anything meaningful. You could achieve the same thing if you used functions that returned strings all the time then did internal deserialization but that's not really a solution. I do get your point about arity and parameter types. You're referring to the fact that not all functions can compose UNLESS they have compatible types and an arity of 1 in the parameters and an arity of 1 in the return value (Golang for example can have arity > 1 in the return value). However this arity issue doesn't really exist. A function with Arity of two is isomorphic to a function of arity 1 that takes in a 2-tuple as input. f(a, b) -> c f'(Tuple[a, b]) -> c g(d) -> Tuple[a, b] Both f' and f are equivalent in theory. Plus you can compose g with f'. The only compatibility problems with composition among functions in the end is just types (not arity) but this problem still exists even in the unix world... it's just hidden from you because you don't actually watch the programs perform deserialization and serialization.
- 6y ago

c3534l 6y ago

The goal of OOP is also to be modular and composable. I think the thing that is a good design choice is modular and composible. It's what the industrial revolution / assembly line was based on. It's what vim keybindings are based on. Heck, it's what programming languages themselves are based on. Here are some simple tools that do easily understandable things together, now put something together with it. Lego bricks are fun and useful and it's not the domain of any one area of CS.

crimsonalucard5 6y ago
>The goal of OOP is also to be modular and composable. Well it depends on the type of composition you're talking about. Anything when mashed together hard enough can compose.

heavyset_go 6y ago

One can say similar things about interfaces in object-oriented programming.

AnimalMuppet 6y ago
Not really. OOP interfaces are usually much more intricate than FP interfaces. You might say that they are less like a lego interface and more like electronic connectors. There are several different electronic connectors types, and they are designed so that you physically can't plug the wrong things together. OOP is like that - a more complicated interface that makes it impossible to connect anything-to-anything the way legos can. On the other hand, if you're trying to build something large (a house, say - a real house, not a toy one), you don't want the lego interface. Sure, you can plug anything to anything, but not all of those connections make sense. Also, for building something that large, legos are too small a building block to be convenient. You want some larger things - beams and sheets of wood and particleboard, pipes, air ducts. You don't want to have to build all of those out of tiny blocks. In the same way, I wonder if building larger applications out of FP is going to be similarly tedious. I have never done it, so I will admit that I don't know.