3 ms·
Related, I find bash and other shells the only language where I can stream data through processes. Like, R can pipe, and pandas can .pipe() but they both compl
by wodenokoto 3y ago
Related, I find bash and other shells the only language where I can stream data through processes.
Like, R can pipe, and pandas can .pipe() but they both complete the function on the entire data before it pushes a copy of the output to the next step.
Why isn’t this flow of data a primitive outside of shell?
- pama 3y agoI am not sure about the design choices of the pandas library or of R, but generally, many programming languages limit the number of processes they spawn during evaluation. Moreover, designing a pipeline that produces a complete output before it is processed by the next function tends to be simpler in these languages. The shell environment is explicit about process creation and offers interactivity, leading to an expectation that a pipeline of processes might not complete before being interrupted. As a result, true pipelining is highly effective in such contexts. Some languages, like Haskell, or certain libraries, employ lazy evaluation to achieve similar outcomes. For instance, by using the polars library instead of pandas, one can achieve true row-by-row pipelining for specific operations in Python.