11 ms·

The Beauty of Unix Pipelines (2020)

97 points by ddtaylor 4y ago

ogogmad 4y ago
If "programs ought to do one thing and do it well", then why aren't they just library functions in a scripting language? Scripting languages can probably do the same things (if given minor syntax improvements, see Xonsh) in one line as well, and scale to have more complicated logic if needed. I still don't see the beauty.
- jdhendrickson 4y ago
  I mean... isn't that what is already happening, the shell is the scripting language, and compiled binaries that mung text are the library functions. Unix/Unixlike operating systems aren't perfect, but they get the job done.
  - dkresge 4y ago
    Moreover, something rarely mentioned, multiple processes and pipes can give you some free parallelism courtesy of the scheduler.
  - glhaynes 4y ago
    The downer is that every function call in this system has only string parameters.
    - unsafecast 4y ago
      That's because this is the way they thought back then. You can trivially pass json data between processes and make a coherent structured system. Nobody does it though.
      - a_conservative 4y ago
        jq has >20k stars on GitHub. I use it mostly for little cli utilities, so maybe it isn’t an exact refutation of your claim. [0] https://github.com/stedolan/jq https://github.com/stedolan/jq
        unsafecast 4y ago
        Sure, but I think the bigger problem is complete systems. Look at a usual Linux box. It has a bunch of files, all with their custom syntax, and you parse them with error-prone shell commands. If everything used json, everyone's lives would've been so much better. Independent programs using json is a step in the right direction though.
- winety 4y ago
  > …then why aren't they just library functions in a scripting language? I’d say they basically are. What’s the difference between Python’s sort and (let’s say) Bash’s sort? In Python I call sort to order a data structure like a list. In Bash I do the same, but on a different data structure. The crux of the matter is probably buried in semantics. > I still don't see the beauty. What I like about shell and pipelines is that they allow me to easily edit and play with text. I don’t need to think about opening nor closing files, I don’t need to think how to best represent the data I’m working with. In minutes I can get the information I need and I can move on.
- indymike 4y ago
  > If "programs ought to do one thing and do it well", then why aren't they just library functions in a scripting language? In many cases, the command line tool is a wrapper for an awesome library, and the awesome library has bindings for multiple scripting languages. So in many cases, there's a command, and a library and yeah, you can import the library into your favorite scripting language (or often compiled language, too). In the end, you could argue that all languages are just an abstraction for system calls anyway. So why the love for the Unix shell? One line is faster for the user than 23 lines. It's the same reason we use compiled languages instead of assembly most of the time. The same applies to using python of javascript instead of C or rust. Less lines of code. Bigger abstractions. Problem is solved and the work is done faster. > I still don't see the beauty. The whole "collection of small programs that do one thing well" thing is just one way to do it, and there are plenty of examples of programs that are super successful that don't heed that advice in any way at all. In the end, Unix has held up for a remarkably long time... seems like there should be a better way to do it, but it's hard to argue with Unix success in the wild.
- dahart 4y ago
  Why would having them just be library functions make it better? (I don’t disagree, just curious what you’re thinking.) The actual differences between pipeable programs and library functions might be hard to pin down concretely. Btw do you use unix & pipes a lot, or not much? Here are a few thoughts, maybe helpful or maybe not. Part of the beauty of pipes and simple lego-brick programs is the complete integration with the operating system, and more specifically the file system. Being able not just to write your state to a file, but furthermore having that be the default mode of operation is pretty powerful, especially combined with special files. Writing to a file in a scripting language that’s not a shell is usually easy, but not the default, and would be cumbersome to do at every step. Part of do one thing well is to keep the shell DRY - don’t add sorting or searching or filtering functionality to another program if you can already use grep or sort or sed. Done right, this helps people stay fluent with a small number of tools. Another implicit benefit of the unix philosophy that wasn’t mentioned here, but does go hand-in-hand with pipes and small-good tools, is the idea to process and work with human readable text files. One way to look at unix beauty is to do some command line data analysis. I like the general philosophy of data exploring with piped commands for one-off and small projects. Sometimes more focused tools are needed, but most of the time perhaps you can get away with unix tools. https://en.wikibooks.org/wiki/Ad_Hoc_Data_Analysis_From_The_Unix_Command_Line https://en.wikibooks.org/wiki/Ad_Hoc_Data_Analysis_From_The_...
- mongol 4y ago
  Or perhaps, why not both? Imagine a monorepo where the same source code that produces the "ls" output also has a library function that returns the same as structs. Perhaps ls is a bad example, but it would be quite neat if the command line and programming context could act on semantically similar procedures. The Rust effort to implement coreutils anew could possibly pick this up and make something really useful here.
- cwalv 4y ago
  You can think of them as library functions, but they all have the same signature: a pair of (byte[]) streams for input and output (plus another for errors), and then `args: string[]` and `options: map<string, string>`. If you're a human that's only really interested in consuming text, this isn't a completely inappropriate interface, and there is some beauty in just how expressive it can be.
- deleted 4y ago
  [deleted]
- astrobe_ 4y ago
  Actually, shell/unix pipes bring to the table a bunch of features: a convenient syntax to connect processes, I/O buffering (usually done by the programs themselves but still), and concurrency. Languages are an (often) unique combination of (often) non-unique features. That's why shell scripting is a thing.
csdvrx 4y ago
It's very nice indeed, but PowerShell pipelines are at another level of beauty: think what you could do if instead of just STDOUT (and STDERR to be fair) you had data structures with names On Windows try for the fun it: Get-ChildItem -Path HKLM:\SYSTEM\ControlSet001\Services\DeviceAssociationService\State\Store\ | Select-String -Pattern Bluetooth Now look at how it parallelize the request by default, and how it returned you several columns of data per line. Want a regex? Get-ChildItem -Path HKLM:\SYSTEM\ControlSet001\Services\DeviceAssociationService\State\Store\ | Select-String -Pattern Bluetooth | ForEach-Object { $_ -replace 'HK.*-','' -replace ':', ''} Want a CSV? Add "| ConvertTo-Csv" Let's do another, to filter only devices (a named property) Get-PnpDevice -Class 'Bluetooth' -PresentOnly | Where-Object HardwareID -match 'DEV_' | Get-PnpDeviceProperty | ConvertTo-Csv Now I'm learning how to do better things like lambdas on the pipeline PowerShell is weird the first time, but it's so well made it'll give you hours and hours of fun :)
- deleted 4y ago
  [deleted]
- Tijdreiziger 4y ago
  The problem with PowerShell (aside from its verbosity) is that you essentially have to relearn the name of every command and every option, throwing all your existing shell knowledge out. I earnestly used PS as my main shell for a number of years, but this learning curve was just too steep. Once WSL turned up I jumped ship.
- xthrowawayxx 4y ago
  Sorrt, I can't tell if this is sarcasm. Is there something I'm missing? Even the above examples look awful to me
  - csdvrx 4y ago
    It's not sarcasm: I actually like PS because it builds on the concept of data pipelines, to turn them into something more object orientend. The only thing similar in the Linux world is using sqlite as the target of pipe to do some data processing, like on a ps output, find the name of the process using the most RAM. Structurally, the language seems both Perl inspired and something a bit like of an hybrid of LISP (for the object) and SQL (for how most structures are manipulated)
hcarvalhoalves 4y ago
> The Unix philosophy lays emphasis on building software that is simple and extensible. Each piece of software must do one thing and do it well. But, you see… his first example shows how the ‘git log’ doesn’t only show the log, but receives a flag to output arbitrary information according to the format string, and ‘uniq’ doesn’t only show unique entries, but receives a flag to count entries. I would say these are not great examples of “do one thing”, and taken literally, the Unix philosophy would require much more shell script gymnastic to do simple tasks.
- uvdn7 4y ago
  Exactly. The definition of “one thing” is unclear. One house is one thing, which also has a thousand things in it. Car is does the “one thing” — driving, but steering is arguably a thing as well.
  - astrobe_ 4y ago
    Don't play dumb. When you can describe the function with one simple sentence, it is likely that it does one thing. Cars transport their passengers. The fact that performing this action involves complex processes doesn't matter. Just like a math theorem can be short and say just one thing and be extremely useful, but its demonstration could require many steps. The idea behind the Unix philosophy is not simplicity of implementation - on the contrary many little Unix tools are complex under the hood because they try to do their thing optimally - but modularity. "Do one thing and do it well" to avoid functional overlap. That's simply an obvious and right thing to do in general. However this is not a principle easy to follow, because of unrelated concerns like stuffing your resume with impressive stuff - the problem with "simple" is that it is often confused with "easy" - or making things than can be sold - simplicity doesn't sell, "powerful" (as in "I have no idea what's the purpose of this does and I don't need that thing, but it's good to have 'just in case'") does.
- pjmlp 4y ago
  That philosophy tends to be cargo cult by those that never used commercial UNIXes. I use UNIX systems since 1993, starting with Xenix, no idea when that started to be something that some people actually believe it happens.
yegle 4y ago
I was wondering if there's a CLI that can replace sort | uniq -c | sort -nr. I feel like this is a very commonly used pattern.
- bvdijfi 4y ago
  Maybe I'm missing something, but what's stopping you from creating a shell alias for exactly that pipeline?
  - jcelerier 4y ago
    The point is that it's slower than an equivalent do-it-all - sort | uniq is already slow for interactive use if you have a couple thousand input lines (with our computers it should absolutely never take more than a few milliseconds)
    - dahart 4y ago
      Who said anything about perf? @yegle said it was a common pattern and asked for alternatives. > it should absolutely never take more than a few milliseconds Why? It surely depends on your use case (highest possible perf is different for many dupes vs few dupes, for example). And a few milliseconds is lower than a human's ability to respond, so doesn't matter for interactive use, it can only matter in batch scenarios. sort | uniq | sort -nr takes ~300ms on a file of 10k lines of random data that is 0.5MB on my 10 year old mac. It's ~60ms on 2k lines. It's faster on a newer computer, and unless you're doing this a thousand times in a row, there's next to no difference between 3 ms or 300ms to me in an interactive shell... BTW, what is the alternative? Are you assuming I write something high performance, spend my time figuring it out? Or is there a higher performance sort|uniq already available?
      - jcelerier 4y ago
        > sort | uniq | sort -nr takes ~300ms on a file of 10k lines of random data that is 0.5MB on my 10 year old mac. It's ~60ms on 2k lines. it's absolutely intolerable > It's faster on a newer computer, and unless you're doing this a thousand times in a row, there's next to no difference between 3 ms or 300ms to me in an interactive shell... having to wait 300ms after an interaction makes my hands shake. If the result is not on screen by the time the enter key has gone back up it's plain bad.
pjmlp 4y ago
I guess one could also refer to the beauty of Lisp REPL, 1961, with the added benefit of having a proper debugging facility and structured data.
menotyou 4y ago
This should remind us what programming is all about: Take a data set, modify it, present it to the user or write it back to a file or a database. And we can see why imperative programming is much more simpler than OOP or functional programming. - That that data! - Transform the data! - Take result of transformation and transform further! - Display the result! Not: - Think about an abstraction of the problem - Define a class which is vaguely near to the problem domain. - Take the data and press into the object using a constructor. - Write some methods which fits the object abstraction but not related to the actual problem - Write a couple of methods to administrate the data inside the object - Do the same for a second class because after doing the first transformation step you have a different kind of object - Use a mixin to combine the two classes - Refactor the code to adhere to Clean Code - Write a helper class which is actually not an abstraction of anything - Write test classes to test quality into your code because your codebase became to messy. - Explain everybody why this approach is superior than just pipelining the data between functions. - Complain about why the state of the industry is so bad.
- unsafecast 4y ago
  I would argue that functional code is much closer to Unix pipelines than imperative code. Each function takes inputs and transforms them. Pipelines are just function composition. 100% agree with everything else you just said though.
  - NackerHughes 4y ago
    I'm glad I'm not the only one who thought this. When I first learned about function composition, a supposedly groundbreaking feature of functional programming, Unix pipes were the first thing that popped into my head, but making that comparison out loud was only met with facepalms and accusations of trolling. But it's literally 'take this input and apply these functions to it, in this order'.
  - menotyou 4y ago
    Yah, I was expecting this comment. I'd agree but I see nowadays to obsession on abstract issues around functional programming like clojures, monads, recursion and code is data and function as first class citizens which might be of theoretical interest, but quite often are only code obfuscating technics. When you use (progn ....) in lisp or (do ...) or (-> ...) in clojure you are pretty near to what I understand as imperative but you are leaving the orthodoxy of functional programming. What is relevant is idempotence which is easily achievable in functional programming and in procedural/imperative as long as you don't use global variables. Idempotence generally not a feature of OOP. (OOP quite often is not even deterministic).
wodenokoto 4y ago
I wish the next data science language had parallel pipelines built in as a default. I love the way I can pipe stuff in R’s tidyverse package, and I’m a big fan of chained pandas, but I wish they streamed data one row at a time and all the pipes ran in parallel. Having that as a primitive in your data science language would be really cool.
- Ratalala 4y ago
  You should check out https://www.pola.rs/ https://www.pola.rs/
mwenge 4y ago
I made https://io10.dev/ https://io10.dev/ to apply this concept to code notebooks.

97 points by ddtaylor 4y ago

If "programs ought to do one thing and do it well", then why aren't they just library functions in a scripting language? Scripting languages can probably do the same things (if given minor syntax improvements, see Xonsh) in one line as well, and scale to have more complicated logic if needed. I still don't see the beauty.

jdhendrickson 4y ago
I mean... isn't that what is already happening, the shell is the scripting language, and compiled binaries that mung text are the library functions. Unix/Unixlike operating systems aren't perfect, but they get the job done.
- dkresge 4y ago
  Moreover, something rarely mentioned, multiple processes and pipes can give you some free parallelism courtesy of the scheduler.
- glhaynes 4y ago
  The downer is that every function call in this system has only string parameters.
  - unsafecast 4y ago
    That's because this is the way they thought back then. You can trivially pass json data between processes and make a coherent structured system. Nobody does it though.
    - a_conservative 4y ago
      jq has >20k stars on GitHub. I use it mostly for little cli utilities, so maybe it isn’t an exact refutation of your claim. [0] https://github.com/stedolan/jq https://github.com/stedolan/jq
      - unsafecast 4y ago
        Sure, but I think the bigger problem is complete systems. Look at a usual Linux box. It has a bunch of files, all with their custom syntax, and you parse them with error-prone shell commands. If everything used json, everyone's lives would've been so much better. Independent programs using json is a step in the right direction though.
winety 4y ago
> …then why aren't they just library functions in a scripting language? I’d say they basically are. What’s the difference between Python’s sort and (let’s say) Bash’s sort? In Python I call sort to order a data structure like a list. In Bash I do the same, but on a different data structure. The crux of the matter is probably buried in semantics. > I still don't see the beauty. What I like about shell and pipelines is that they allow me to easily edit and play with text. I don’t need to think about opening nor closing files, I don’t need to think how to best represent the data I’m working with. In minutes I can get the information I need and I can move on.
indymike 4y ago
> If "programs ought to do one thing and do it well", then why aren't they just library functions in a scripting language? In many cases, the command line tool is a wrapper for an awesome library, and the awesome library has bindings for multiple scripting languages. So in many cases, there's a command, and a library and yeah, you can import the library into your favorite scripting language (or often compiled language, too). In the end, you could argue that all languages are just an abstraction for system calls anyway. So why the love for the Unix shell? One line is faster for the user than 23 lines. It's the same reason we use compiled languages instead of assembly most of the time. The same applies to using python of javascript instead of C or rust. Less lines of code. Bigger abstractions. Problem is solved and the work is done faster. > I still don't see the beauty. The whole "collection of small programs that do one thing well" thing is just one way to do it, and there are plenty of examples of programs that are super successful that don't heed that advice in any way at all. In the end, Unix has held up for a remarkably long time... seems like there should be a better way to do it, but it's hard to argue with Unix success in the wild.
dahart 4y ago
Why would having them just be library functions make it better? (I don’t disagree, just curious what you’re thinking.) The actual differences between pipeable programs and library functions might be hard to pin down concretely. Btw do you use unix & pipes a lot, or not much? Here are a few thoughts, maybe helpful or maybe not. Part of the beauty of pipes and simple lego-brick programs is the complete integration with the operating system, and more specifically the file system. Being able not just to write your state to a file, but furthermore having that be the default mode of operation is pretty powerful, especially combined with special files. Writing to a file in a scripting language that’s not a shell is usually easy, but not the default, and would be cumbersome to do at every step. Part of do one thing well is to keep the shell DRY - don’t add sorting or searching or filtering functionality to another program if you can already use grep or sort or sed. Done right, this helps people stay fluent with a small number of tools. Another implicit benefit of the unix philosophy that wasn’t mentioned here, but does go hand-in-hand with pipes and small-good tools, is the idea to process and work with human readable text files. One way to look at unix beauty is to do some command line data analysis. I like the general philosophy of data exploring with piped commands for one-off and small projects. Sometimes more focused tools are needed, but most of the time perhaps you can get away with unix tools. https://en.wikibooks.org/wiki/Ad_Hoc_Data_Analysis_From_The_Unix_Command_Line https://en.wikibooks.org/wiki/Ad_Hoc_Data_Analysis_From_The_...
mongol 4y ago
Or perhaps, why not both? Imagine a monorepo where the same source code that produces the "ls" output also has a library function that returns the same as structs. Perhaps ls is a bad example, but it would be quite neat if the command line and programming context could act on semantically similar procedures. The Rust effort to implement coreutils anew could possibly pick this up and make something really useful here.
cwalv 4y ago
You can think of them as library functions, but they all have the same signature: a pair of (byte[]) streams for input and output (plus another for errors), and then `args: string[]` and `options: map<string, string>`. If you're a human that's only really interested in consuming text, this isn't a completely inappropriate interface, and there is some beauty in just how expressive it can be.
deleted 4y ago
[deleted]
astrobe_ 4y ago
Actually, shell/unix pipes bring to the table a bunch of features: a convenient syntax to connect processes, I/O buffering (usually done by the programs themselves but still), and concurrency. Languages are an (often) unique combination of (often) non-unique features. That's why shell scripting is a thing.

csdvrx 4y ago

It's very nice indeed, but PowerShell pipelines are at another level of beauty: think what you could do if instead of just STDOUT (and STDERR to be fair) you had data structures with names On Windows try for the fun it: Get-ChildItem -Path HKLM:\SYSTEM\ControlSet001\Services\DeviceAssociationService\State\Store\ | Select-String -Pattern Bluetooth Now look at how it parallelize the request by default, and how it returned you several columns of data per line. Want a regex? Get-ChildItem -Path HKLM:\SYSTEM\ControlSet001\Services\DeviceAssociationService\State\Store\ | Select-String -Pattern Bluetooth | ForEach-Object { $_ -replace 'HK.*-','' -replace ':', ''} Want a CSV? Add "| ConvertTo-Csv" Let's do another, to filter only devices (a named property) Get-PnpDevice -Class 'Bluetooth' -PresentOnly | Where-Object HardwareID -match 'DEV_' | Get-PnpDeviceProperty | ConvertTo-Csv Now I'm learning how to do better things like lambdas on the pipeline PowerShell is weird the first time, but it's so well made it'll give you hours and hours of fun :)

deleted 4y ago
[deleted]
Tijdreiziger 4y ago
The problem with PowerShell (aside from its verbosity) is that you essentially have to relearn the name of every command and every option, throwing all your existing shell knowledge out. I earnestly used PS as my main shell for a number of years, but this learning curve was just too steep. Once WSL turned up I jumped ship.
xthrowawayxx 4y ago
Sorrt, I can't tell if this is sarcasm. Is there something I'm missing? Even the above examples look awful to me
- csdvrx 4y ago
  It's not sarcasm: I actually like PS because it builds on the concept of data pipelines, to turn them into something more object orientend. The only thing similar in the Linux world is using sqlite as the target of pipe to do some data processing, like on a ps output, find the name of the process using the most RAM. Structurally, the language seems both Perl inspired and something a bit like of an hybrid of LISP (for the object) and SQL (for how most structures are manipulated)

hcarvalhoalves 4y ago

> The Unix philosophy lays emphasis on building software that is simple and extensible. Each piece of software must do one thing and do it well. But, you see… his first example shows how the ‘git log’ doesn’t only show the log, but receives a flag to output arbitrary information according to the format string, and ‘uniq’ doesn’t only show unique entries, but receives a flag to count entries. I would say these are not great examples of “do one thing”, and taken literally, the Unix philosophy would require much more shell script gymnastic to do simple tasks.

uvdn7 4y ago
Exactly. The definition of “one thing” is unclear. One house is one thing, which also has a thousand things in it. Car is does the “one thing” — driving, but steering is arguably a thing as well.
- astrobe_ 4y ago
  Don't play dumb. When you can describe the function with one simple sentence, it is likely that it does one thing. Cars transport their passengers. The fact that performing this action involves complex processes doesn't matter. Just like a math theorem can be short and say just one thing and be extremely useful, but its demonstration could require many steps. The idea behind the Unix philosophy is not simplicity of implementation - on the contrary many little Unix tools are complex under the hood because they try to do their thing optimally - but modularity. "Do one thing and do it well" to avoid functional overlap. That's simply an obvious and right thing to do in general. However this is not a principle easy to follow, because of unrelated concerns like stuffing your resume with impressive stuff - the problem with "simple" is that it is often confused with "easy" - or making things than can be sold - simplicity doesn't sell, "powerful" (as in "I have no idea what's the purpose of this does and I don't need that thing, but it's good to have 'just in case'") does.
pjmlp 4y ago
That philosophy tends to be cargo cult by those that never used commercial UNIXes. I use UNIX systems since 1993, starting with Xenix, no idea when that started to be something that some people actually believe it happens.

yegle 4y ago

I was wondering if there's a CLI that can replace sort | uniq -c | sort -nr. I feel like this is a very commonly used pattern.

bvdijfi 4y ago
Maybe I'm missing something, but what's stopping you from creating a shell alias for exactly that pipeline?
- jcelerier 4y ago
  The point is that it's slower than an equivalent do-it-all - sort | uniq is already slow for interactive use if you have a couple thousand input lines (with our computers it should absolutely never take more than a few milliseconds)
  - dahart 4y ago
    Who said anything about perf? @yegle said it was a common pattern and asked for alternatives. > it should absolutely never take more than a few milliseconds Why? It surely depends on your use case (highest possible perf is different for many dupes vs few dupes, for example). And a few milliseconds is lower than a human's ability to respond, so doesn't matter for interactive use, it can only matter in batch scenarios. sort | uniq | sort -nr takes ~300ms on a file of 10k lines of random data that is 0.5MB on my 10 year old mac. It's ~60ms on 2k lines. It's faster on a newer computer, and unless you're doing this a thousand times in a row, there's next to no difference between 3 ms or 300ms to me in an interactive shell... BTW, what is the alternative? Are you assuming I write something high performance, spend my time figuring it out? Or is there a higher performance sort|uniq already available?
    - jcelerier 4y ago
      > sort | uniq | sort -nr takes ~300ms on a file of 10k lines of random data that is 0.5MB on my 10 year old mac. It's ~60ms on 2k lines. it's absolutely intolerable > It's faster on a newer computer, and unless you're doing this a thousand times in a row, there's next to no difference between 3 ms or 300ms to me in an interactive shell... having to wait 300ms after an interaction makes my hands shake. If the result is not on screen by the time the enter key has gone back up it's plain bad.

pjmlp 4y ago

I guess one could also refer to the beauty of Lisp REPL, 1961, with the added benefit of having a proper debugging facility and structured data.

menotyou 4y ago

This should remind us what programming is all about: Take a data set, modify it, present it to the user or write it back to a file or a database. And we can see why imperative programming is much more simpler than OOP or functional programming. - That that data! - Transform the data! - Take result of transformation and transform further! - Display the result! Not: - Think about an abstraction of the problem - Define a class which is vaguely near to the problem domain. - Take the data and press into the object using a constructor. - Write some methods which fits the object abstraction but not related to the actual problem - Write a couple of methods to administrate the data inside the object - Do the same for a second class because after doing the first transformation step you have a different kind of object - Use a mixin to combine the two classes - Refactor the code to adhere to Clean Code - Write a helper class which is actually not an abstraction of anything - Write test classes to test quality into your code because your codebase became to messy. - Explain everybody why this approach is superior than just pipelining the data between functions. - Complain about why the state of the industry is so bad.

unsafecast 4y ago
I would argue that functional code is much closer to Unix pipelines than imperative code. Each function takes inputs and transforms them. Pipelines are just function composition. 100% agree with everything else you just said though.
- NackerHughes 4y ago
  I'm glad I'm not the only one who thought this. When I first learned about function composition, a supposedly groundbreaking feature of functional programming, Unix pipes were the first thing that popped into my head, but making that comparison out loud was only met with facepalms and accusations of trolling. But it's literally 'take this input and apply these functions to it, in this order'.
- menotyou 4y ago
  Yah, I was expecting this comment. I'd agree but I see nowadays to obsession on abstract issues around functional programming like clojures, monads, recursion and code is data and function as first class citizens which might be of theoretical interest, but quite often are only code obfuscating technics. When you use (progn ....) in lisp or (do ...) or (-> ...) in clojure you are pretty near to what I understand as imperative but you are leaving the orthodoxy of functional programming. What is relevant is idempotence which is easily achievable in functional programming and in procedural/imperative as long as you don't use global variables. Idempotence generally not a feature of OOP. (OOP quite often is not even deterministic).

wodenokoto 4y ago

I wish the next data science language had parallel pipelines built in as a default. I love the way I can pipe stuff in R’s tidyverse package, and I’m a big fan of chained pandas, but I wish they streamed data one row at a time and all the pipes ran in parallel. Having that as a primitive in your data science language would be really cool.

Ratalala 4y ago
You should check out https://www.pola.rs/ https://www.pola.rs/

mwenge 4y ago

I made https://io10.dev/ https://io10.dev/ to apply this concept to code notebooks.