14 ms·

Pipelining might be my favorite programming language feature

• SimonDorfman 1y ago
  The tidyverse folks in R have been using that for a while: https://magrittr.tidyverse.org/reference/pipe.html https://magrittr.tidyverse.org/reference/pipe.html
  • flobosg 1y ago
    Base R as well: |> was implemented as a pipe operator in 4.1.0.
    • tylermw 1y ago
      Importantly, the base R pipe implements the operation at the language parsing level, so it has basically zero overhead.
      • zelphirkalt 1y ago
        I would assume, that most languages do that, or alternatively have a compiler, that is smart enough to ensure there is no actual overhead in the compiled code.
  • madcaptenor 1y ago
    And base R has had a pipe for a couple years now, although there are some differences between base R's |> and tidyverse's %>%: https://www.tidyverse.org/blog/2023/04/base-vs-magrittr-pipe/ https://www.tidyverse.org/blog/2023/04/base-vs-magrittr-pipe...
  • thom 1y ago
    I've always found magrittr mildly hilarious. R has vestigial Lisp DNA, but somehow the R implementation of pipes was incredibly long, complex and produced stack traces, so it moved to a native C implementation, which nevertheless has to manipulate the SEXPs that secretly underlie the language. Compared to something like Clojure's threading macros it's wild how much work is needed.
  • steine65 1y ago
    R, specifically tidyverse, has a special place in my heart. Tidy principles makes data analysis easy to read and easy to use new functions, since there are standards that must be met to call a function "tidy." Recently I started using Nushell, which feels very similar.
  • mvieira38 1y ago
    R + tidyverse is the gold standard for working with data quickly in a readable and maintainable way, IMO. It's just absolutely seamless. Shoutout to tidyverts (https://tidyverts.org/ https://tidyverts.org/) for working with time series, too
• jaymbo 1y ago
  This is why I love Scala so much
  • minebreaker 1y ago
    `tap` is cool.
  • rad_gruchalski 1y ago
    Scala is by far one of the nicest programming languages I have ever worked with. Scala with no JVM dependency would a killer programming language BUT only when all async features work out of the box like they do JVM. It’s been attempted a couple of times and it never succeeded.
    • threeseed 1y ago
      a) Scala with no JVM exists: https://scala-native.org https://scala-native.org or https://www.scala-js.org https://www.scala-js.org b) Async works on Scala Native: https://github.com/lampepfl/gears https://github.com/lampepfl/gears and is coming to Scala.js.
• shae 1y ago
  If Python object methods returned `self` by default instead of `None` you could do this in Python too! This is my biggest complaint about Python.
  • incognito124 1y ago
    there are https://github.com/JulienPalard/Pipe https://github.com/JulienPalard/Pipe and https://github.com/0101/pipetools https://github.com/0101/pipetools
• mexicocitinluez 1y ago
  LINQ is easily one of C#'s best features.
  • adzm 1y ago
    Interestingly though the actual integrated query part is much less useful or widely used as the methods on IEnumerable etc.
• kordlessagain 1y ago
  While the author claims "semantics beat syntax every day of the week," the entire article focuses on syntax preferences rather than semantic differences. Pipelining can become hard to debug when chains get very long. The author doesn't address how hard it can be to identify which step in a long chain caused an error. They do make fun of Python, however. But don't say much about why they don't like it other than showing a low-res photo of a rock with a pipe routed around it. Ambiguity about what constitutes "pipelining" is the real issue here. The definition keeps shifting throughout the article. Is it method chaining? Operator overloading? First-class functions? The author uses examples that function very differently.
  • pavel_lishin 1y ago
    The article also clearly points that that it's just a hot-take, and to not take it too seriously.
  • zelphirkalt 1y ago
    You can add peek steps in pipelines and inspect the in between results. Not really any different from normal function call debugging imo.
    • krapht 1y ago
      Yes, but here's my hot take - what if you didn't have to edit the source code to debug it? Instead of chaining method calls you just assign to a temporary variable. Then you can set breakpoints and inspect variable values like you do normally without editing source. It's not like you lose that much readability from foo(bar(baz(c))) c |> baz |> bar |> foo c.baz().bar().foo() t = c.baz() t = t.bar() t = t.foo()
      • Mond_ 1y ago
        I feel like a sufficiently good debugger should allow you to place a breakpoint at any of the lines here, and it should break exactly at that specific line. fn get_ids(data: Vec<Widget>) -> Vec<Id> { data.iter() .filter(|w| w.alive) .map(|w| w.id) .collect() } It sounds to me like you're asking for linebreaks. Chaining doesn't seem to be the issue here.
• kuon 1y ago
  That's also why I enjoy elixir a lot. The |> operator is really cool.
• drchickensalad 1y ago
  I miss F#
  • aloisdg 1y ago
    So do I sibling. so do I
• osigurdson 1y ago
  C# has had "Pipelining" (aka Linq) for 17 years. I do miss this kind of stuff in Go a little.
  • vjvjvjvjghv 1y ago
    Agreed. It would be nice if SQL databases supported something similar.
    • NortySpock 1y ago
      I've used "a series of CTEs" to apply a series of transformations and filters, but it's not nearly as elegant as the pipe syntax.
    • sidpatil 1y ago
      PRQL [1] is a pipeline-based query language that compiles to SQL. [1] https://prql-lang.org/ https://prql-lang.org/
  • bob1029 1y ago
    I don't see how LINQ provides an especially illuminating example of what is effectively method chaining. It is an exemplar of expressions [0] more than anything else, which have little to do with the idea of passing results from one method to another. [0]: https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/language-specification/expressions https://learn.microsoft.com/en-us/dotnet/csharp/language-ref...
    • delusional 1y ago
      You might be talking about LINQ queries, while the person you are responding to is probably talking about LINQ in Method Syntax[1] [1]: https://learn.microsoft.com/en-us/dotnet/csharp/linq/get-started/write-linq-queries#standard-query-operator-extension-methods https://learn.microsoft.com/en-us/dotnet/csharp/linq/get-sta...
    • hahn-kev 1y ago
      So many things have been called Linq over the years it's hard to talk about at this point. I've written C# for many years now and I'm not even sure what I would say it's referring to, so I avoid the term. In this case I would say extension methods are what he's really referring to, of which Linq to objects is built on top of.
• zelphirkalt 1y ago
  To one up this: Of course it is even better, if your language allows you to implement proper pipelining with implicit argument passing by yourself. Then the standard language does not need to provide it and assign meaning to some symbols for pipelining. You can decide for yourself what symbols are used and what you find intuitive. Pipelining can guide one to write a bit cleaner code, viewing steps of computation as such, and not as modifications of global state. It forces one to make each step return a result, write proper functions. I like proper pipelining a lot.
  • Mond_ 1y ago
    > if your language allows you to implement proper pipelining with implicit argument passing by yourself > You can decide for yourself what symbols are used and what you find intuitive i mean this sounds fun but tbh it also sounds like it'd result in my colleague Carl defining an utterly bespoke DSL in the language, and using it to write the worst spaghetti code the world has ever seen, leaving the code base an unreadable mess full of sharp edges and implicit behavior
    • 0x1ceb00da 1y ago
      Even veterans mess things up if you use too much of these exotic syntaxes. For loops and if statements rock, but they aren't cool and functional so they aren't discussed much.
• mrkeen 1y ago
  data.iter() .filter(|w| w.alive) .map(|w| w.id) .collect() collect(map(filter(iter(data), |w| w.alive), |w| w.id)) The second approach is open for extension - it allows you to write new functions on old datatypes. > Quick challenge for the curious Rustacean, can you explain why we cannot rewrite the above code like this, even if we import all of the symbols? Probably for lack of > weird operators like <$>, <*>, $, or >>=
  • esafak 1y ago
    Extension methods to the rescue: https://en.wikipedia.org/wiki/Extension_method https://en.wikipedia.org/wiki/Extension_method Examples: https://kotlinlang.org/docs/extensions.html https://kotlinlang.org/docs/extensions.html https://docs.scala-lang.org/scala3/reference/contextual/extension-methods.html https://docs.scala-lang.org/scala3/reference/contextual/exte... See also: https://en.wikipedia.org/wiki/Uniform_function_call_syntax https://en.wikipedia.org/wiki/Uniform_function_call_syntax
    • vips7L 1y ago
      I really wish you couldn't write extensions on nullable types. It's confusing to be able to call what look like instance functions on something clearly nullable without checking. fun main() { val s: String? = null println(s.isS()) // false } fun String?.isS() = "s" == this
      • usrusr 1y ago
        The difference between .let{} and ?.let{} has great utility. You'd either have to give that up or promote let from regular code in the standard library to magic language feature. And you'd lose all those cases of extension methods where the convenience of accepting null left of the dot is their sole reason to be. Null is a valid state, not something incredibly scary best dealt with with a full reboot or better yet throwing away the container. Kotlin is about making peace with null, instead of pretending that null does not exist. (yes, I'm looking at you, Scala) What I do agree with is that extension methods should be a last ditch solution. I'd actually like to see a way to do the nullable receiver thing defined more like regular functions. Perhaps something like fun? name() = if (this==null) "(Absent)" else this.name that is defined inside the regular class block, imported like a regular method (as part of the class) and even present in the class object e.g. for reflection on the non-null case (and for Java compat where that still matters)
• epolanski 1y ago
  I personally like how effect-ts allows you to write both pipelines or imperative code to express the very same things. Building pipelines: https://effect.website/docs/getting-started/building-pipelines/ https://effect.website/docs/getting-started/building-pipelin... Using generators: https://effect.website/docs/getting-started/using-generators/ https://effect.website/docs/getting-started/using-generators... Having both options is great (at the beginning effect had only pipe-based pipelines), after years of writing effect I'm convinced that most of the time you'd rather write and read imperative code than pipelines which definitely have their place in code bases. In fact most of the community, at large, converged at using imperative-style generators over pipelines and having onboarded many devs and having seen many long-time pipeliners converging to classical imperative control flow seems to confirm both debugging and maintenance seem easier.
• bnchrch 1y ago
  I'm personally someone who advocates for languages to keep their feature set small and shoot to achieve a finished feature set quickly. However. I would be lying if I didn't secretly wish that all languages adopted the `|>` syntax from Elixir. ``` params |> Map.get("user") |> create_user() |> notify_admin() ```
  • valenterry 1y ago
    I prefer Scala. You can write ``` params.get("user") |> create_user |> notify_admin ``` Even more concise and it doesn't even require a special language feature, it's just regular syntax of the language ( |> is a method like .get(...) so you could even write `params.get("user").|>(create_user) if you wanted to)
    • elbasti 1y ago
      In elixir, ```Map.get("user") |> create_user |> notify_admin ``` would aso be valid, standard elixir, just not idiomatic (parens are optional, but preferred in most cases, and one-line pipes are also frowned upon except for scripting).
      • MaxBarraclough 1y ago
        With the disclaimer that I don't know Elixir and haven't programmed with the pipeline operator before: I don't like that special () syntax. That syntax denotes application of the function without passing any arguments, but the whole point here is that an argument is being passed. It seems clearer to me to just put the pipeline operator and the name of the function that it's being used with. I don't see how it's unclear that application is being handled by the pipeline operator. Also, what if the function you want to use is returned by some nullary function? You couldn't just do |> getfunc(), as presumably the pipeline operator will interfere with the usual meaning of the parentheses and will try to pass something to getfunc. Would |> ( getfunc() ) work? This is the kind of problem that can arise when one language feature is permitted to change the ordinary behaviour of an existing feature in the name of convenience. (Unless of course I'm just missing something.)
• chewbacha 1y ago
  Is this pipelining or the builder pattern?
  • meltyness 1y ago
    Pipes and filters are considered an architectural pattern, whereas Builder is a GoF OOP pattern, so yes.
  • Mond_ 1y ago
    "These are the same picture." (Sort of.)
  • ivanjermakov 1y ago
    I usually call it method chaining. Where the builder pattern use it.
• cutler 1y ago
  Clojure has pipeline functions -> and ->> without resorting to OO dot syntax.
  • jolt42 1y ago
    As well as some-> (exit on null) and cond-> (with predicates) that are often handy.
    • joeevans1000 1y ago
      As well as a lot of flexibility on where the result of the previous step feeds into the current one.
    • 3036e4 1y ago
      Both Fennel and Janet has the Clojure threading macros, with -?> and -?>> for false/null checks, but not any other variants as far as I know.
• deleted 1y ago
  [deleted]
• amelius 1y ago
  Am I the only one who thinks yuck? Instead of writing: a().b().c().d(), it's much nicer to write: d(c(b(a()))), or perhaps (d ∘ c ∘ b ∘ a)().
  • vinceguidry 1y ago
    Why, if you don't have to, would you write the functions in reverse order of when they're applied?
    • gloxkiqcza 1y ago
      Presumably because they’ve been doing so for decades so it seems logical and natural in their head while the new thing is new and thus unintuitive.
    • dboreham 1y ago
      Because it makes more sense?
      • Mond_ 1y ago
        Does it actually make more sense, or is it just more familiar?
      • kortex 1y ago
        I would wager within a rounding error, all humans have a lifetime of experience in following directions of the form: 1. do the first step in the process 2. then do the next thing 3. followed by a third action I struggle to think of any context outside of programming, retrosynthesis in chemistry, and some aspects of reverse-Polish notation calculators, where you conceive of the operations/arguments last-to-first. All of which are things typically encountered pretty late in one's educational journey.
        • amelius 1y ago
          Consistency is more important. If you ever wrote: a(b()) then you're already breaking your left-to-right/first-to-last rule.
• 1899-12-30 1y ago
  You can somewhat achieve a pipelined like system in sql by breaking down your steps into multiple CTEs. YMMV on the performance though.
  • infogulch 1y ago
    Yeah, the way to get logical pipelining in SQL without CTEs is nested subqueries in the FROM clause. Unfortunately, the nesting is syntactically ugly and confusing to read which is basically the whole idea behind pipeline syntax.
• singularity2001 1y ago
  I tried to convince the julia authors to make a.b(c) synonymous to b(a,c) like in nim (for similar reasons as in the article). They didn't like it.
  • queuebert 1y ago
    What were their reasons?
    • pansa2 1y ago
      I suspect: Julia's multiple dispatch means that all arguments to a function are treated equally. The syntax `b(a, c)` makes this clear, whereas `a.b(c)` makes it look like `a` is in some way special.
  • sparkie 1y ago
    I don't like it either, because it promotes method `b` to the global namespace. There may be many such `b` methods on different, unrelated types. I think that the latter should be prefixed with the typename or module name. a.b(c) == AType.b(a, c) (or AType::b(a, c) , C++ style)
    • singularity2001 1y ago
      It's the other way around: in Julia b are functions which are globally visible by default and I just suggested to optionally hide them or find them via the object a.
• dapperdrake 1y ago
  Pipelining in software is covered by Richard C. Waters (1989a, 1989b). Wrangles this library to work with JavaScript. Incredibly effective. Much faster at writing and composing code. And this code executes much faster. https://dspace.mit.edu/handle/1721.1/6035 https://dspace.mit.edu/handle/1721.1/6035 https://dspace.mit.edu/handle/1721.1/6031 https://dspace.mit.edu/handle/1721.1/6031 https://dapperdrake.neocities.org/faster-loops-javascript.html https://dapperdrake.neocities.org/faster-loops-javascript.ht...
• blindseer 1y ago
  This article is great, and really distills why the ergonomics of Rust is so great and why languages like Julia are so awful in practice.
  • jakobnissen 1y ago
    You mean tab completion in Rust? Otherwise, let me introduce you to: imap(f) = x -> Iterators.map(f, x) ifilter(f) = x -> Iterators.filter(f, x) v = things |> ifilter(isodd) |> imap(do_process) |> collect
• hliyan 1y ago
  I always wondered how programming would be if we hadn't designed the assignment operator to be consistent with mathematics, and instead had it go LHS -> RHS, i.e. you perform the operation and then decide its destination, much like Unix pipes.
  • RodgerTheGreat 1y ago
    Plenty of LTR languages to choose from, especially concatenative languages like Forth, Joy, or Factor. The APL family is similarly consistent, except RTL.
  • donatj 1y ago
    TI-BASIC is like this with its store operator →. I always liked it. 10→A A+10→C
    • recursive 1y ago
      It also has an = operator, which saves the whole expression. It re-evaluates it then every time it was used.
      • donatj 1y ago
        Yep, CAS right in the BASIC on the 89 and up is a truly magical experience
  • remram 1y ago
    For function calls too? List the arguments then the function's name?
• TrianguloY 1y ago
  Kotlin sort of have it with let (and run) a().let{ b(it) }.let{ c(it) }
  • hombre_fatal 1y ago
    Yeah, Kotlin's solution is nice because it's so general: you can chain on to anything instead of needing everyone to implement a builder pattern. And it's already idiomatic unlike bolting a pipeline operator onto a language that didn't start with it.
    • jillesvangurp 1y ago
      If you see somebody using a builder in Kotlin, they're basically doing it wrong. You can usually get rid of that stuff with a 1 line extension function (for example if it's some Java API that's being called). // extension function on Foo.Companion (similar to static class function in Java) fun Foo.Companion.create(block: FooBuilder.() -> Unit): Foo = FooBuilder().apply(block).build() // example usage val myFoo = Foo.create { setSomeproperty("foo") setAnotherProperty("bar") } Works for any Java/Kotlin API that forces you into method chaining and calling build() manually. Also works without extension functions. You can just call it fun createAFoo(..) or whatever. Looking around in the Kotlin stdlib code base is instructive. Lots of little 1/2 liners like this.
• wslh 1y ago
  I also like a syntax that includes pipelining parallelization, for example: A .B .C || D || E
  • regular_trash 1y ago
    Wouldn't this complicate variable binding? I'm unsure how to think about this kinda of syntax if either D or E are expected to return some kind of data instead of "fire and forget" processes.
• tantalor 1y ago
  > allows you to omit a single argument from your parameter list, by instead passing the previous value I have no idea what this is trying to say, or what it has to do with the rest of the article.
  • delusional 1y ago
    It's getting at the essential truth that for all(?) mainstream languages since object orientation and the dot syntax became a thing `a.b()` implicitly includes `a` as the first argument to the actual method `b(a self)`. Different languages have different constructs on top of that, C++ for example includes a virtual dispatch mechanism, but the one common idea of the _method call_ is that the `self` pointer is passed as the first argument.
• 0xf00ff00f 1y ago
  First example doesn't look bad in C++23: auto get_ids(std::span<const Widget> data) { return data | filter(&Widget::alive) | transform(&Widget::id) | to<std::vector>(); }
  • inetknght 1y ago
    This is not functionally different from operator<< which std::cout has taught us is a neat trick but generally a bad idea.
    • senderista 1y ago
      Unlike the iostreams shift operators, the ranges pipe operator isn't stateful.
      • jjmarr 1y ago
        There's state when you try to use the final result, though. It's not threadsafe due to caching. https://www.youtube.com/watch?v=c1gfbbE2zts https://www.youtube.com/watch?v=c1gfbbE2zts
  • Shorel 1y ago
    This looks awesome! I'm really want to start playing with some C++23 in the future.
    • 0xf00ff00f 1y ago
      I cheated a bit, I omitted the namespaces. Here's a working version: https://godbolt.org/z/1rE9o3Y95 https://godbolt.org/z/1rE9o3Y95
  • uzerfcwn 1y ago
    To me, the cool (and uncommon in other languages' standard libraries) part about C++ ranges is that they reify pipelines so that you can cut and paste them into variables, like so: auto get_ids(std::span<const Widget> data) { auto pipeline = filter(&Widget::alive) | transform(&Widget::id); auto sink = to<std::vector>(); return data | pipeline | sink; }
• RHSeeger 1y ago
  I feel like, at least in some cases, the article is going out of its way to make the "undesired" look worse than it needs to be. Compairing fn get_ids(data: Vec<Widget>) -> Vec<Id> { collect(map(filter(map(iter(data), |w| w.toWingding()), |w| w.alive), |w| w.id)) } to fn get_ids(data: Vec<Widget>) -> Vec<Id> { data.iter() .map(|w| w.toWingding()) .filter(|w| w.alive) .map(|w| w.id) .collect() } The first one would read more easily (and, since it called out, diff better) fn get_ids(data: Vec<Widget>) -> Vec<Id> { collect( map( filter( map(iter(data), |w| w.toWingding()), |w| w.alive), |w| w.id)) } Admittedly, the chaining is still better. But a fair number of the article's complaints are about the lack of newlines being used; not about chaining itself.
  • the_sleaze_ 1y ago
    In my eyes newlines don't solve what I feel to be the issue. Reader needs to recognize reading from left->right to right->left. Of course this really only matters when you're 25 minutes into critical downtime and a bug is hiding somewhere in these method chains. Anything that is surprising needs to go. IMHO it would be better to set intermediate variables with dead simple names instead of newlines. fn get_ids(data: Vec<Widget>) -> Vec<Id> { let iter = iter(data); let wingdings = map(iter, |w| w.toWingding()); let alive_wingdings = filter(wingdings, |w| w.alive); let ids = map(alive_wingdings, |w| w.id); let collected = collect(ids); collected }
    • trealira 1y ago
      > Reader needs to recognize reading from left->right to right->left. Yeah, I agree. The problem is that you have to keep track of nesting in the middle of the expression and then unnest it at the end, which is taxing. So, I also think it could also read better written like this, with the arguments reversed, so you don't have to read it both ways: fn get_ids(data: Vec<Widget>) -> Vec<Id> { collect( map(|w| w.id, filter |w| w.alive, (map(|w| w.toWingding(), iter(data))))) } That's also what they do in Haskell. The first argument to map is the mapping function, the first argument to filter is the predicate function, and so on. People will often just write the equivalent of: getIDs = map getID . filter alive . map toWingDing as their function definitions, with the argument omitted because using the function composition operator looks neater than using a bunch of dollar signs or parentheses. Making it the second argument only makes sense when functions are written after their first argument, not before, to facilitate writing "foo.map(f).filter(y)".
• gus_leonel 1y ago
  [dead]
• guerrilla 1y ago
  This is just super basic functional programming. Seems like we're taking the long way around...
  • Mond_ 1y ago
    Have you read the article? This isn't about functional vs. imperative programming, it's (if anything) about two different ways to write functional code.
    • guerrilla 1y ago
      Keywords "super basic". You learn this in a "my first Haskell" tutorials. Seems tortured in whatever language that is though.
• duped 1y ago
  A pipeline operator is just partial application with less power. You should be able to bind any number of arguments to any places in order to create a new function and "pipe" its output(s) to any other number of functions. One day, we'll (re)discover that partial application is actually incredibly useful for writing programs and (non-Haskell) languages will start with it as the primitive for composing programs instead of finding out that it would be nice later, and bolting on a restricted subset of the feature.
  • choult 1y ago
    ... and then recreate the scripting language...
    • stogot 1y ago
      I was just thinking does this not sound like a shell language? Using | instead of .function()
  • dayvigo 1y ago
    Sure. But how do you write that in a way that is expressive, terse, and readable all at once? Nothing beats x | y | z or (-> x y z). The speed of both writing and reading (and comprehending), the sheer simplicity, is what makes pipelining useful in the first place.
  • zelphirkalt 1y ago
    I like partial application like in Standard ML, but it also means, that one must be very careful with the order of arguments, unless we get a variant of partial application, that is flexible enough to let you specify which arguments you want to provide, instead of always assuming the first n arguments. I use "cut" for this in Scheme. Threading/Pipelines are still very useful though and can shorten things and make them very readable.
  • gpderetta 1y ago
    for loops are also gotos with less power, yet we usually prefer them.
• weinzierl 1y ago
  I suffer from (what I call) bracket claustrophobia. Whenever brackets get nested too deep I makes me uncomfortable. But I fully realize that there are people who are the complete opposite. Lisp programmers are apparently as claustrophil as cats and spelunkers.
  • monsieurbanana 1y ago
    Forget the parenthesis, embrace the automatic indentation and code source manipulations that only perfectly balanced homoiconic expressions can give you.
• pxc 1y ago
  Maybe it's because I love the Unix shell environment so much, but I also really love this style. I try to make good use of it in every language I write code in, and I think it helps make my control flow very simple. With lots of pipelines, and few conditionals or loops, everything becomes very easy to follow.
• taeric 1y ago
  A thing I really like about pipelines in shell scripts, is all of the buffering and threading implied by them. Semantically, you can see what command is producing output, and what command is consuming it. With some idea of how the CPU will be split by them. This is far different than the pattern described in the article, though. Small shame they have come to have the same name. I can see how both work with the metaphor; such that I can't really complain. The "pass a single parameter" along is far less attractive to me, though.
• tpoacher 1y ago
  pipelines are great IF you can easily debug them as easily as temp variable assignments ... looking at you R and tidyverse hell.
• layer8 1y ago
  The one thing that I don’t like about pipelining (whether using a pipe operator or method chaining), is that assigning the result to a variable goes in the wrong direction, so to speak. There should be an equivalent of the shell’s `>` for piping into a variable as the final step. Of course, if the variable is being declared at the same time, whatever the concrete syntax is would still require some getting used to, being “backwards” compared to regular assignment/initialization.
  • jiggunjer 1y ago
    Exists in R: Mydata %>% myfun -> myresult
  • AndyKluger 1y ago
    FWIW in Factor you can set dynamic variables with "coolvalue" thisisthevar set or if you use the `variables` vocab, alternately: "coolvalue" set: thisisthevar and lexical variables are set with "coolvalue" :> thisisthevar
• bluSCALE4 1y ago
  Same. The sad part is that pipelining seems to be something AI is really good at so I'm finding myself writing less of it.
• joeevans1000 1y ago
  Clojure threading, of course.
• flakiness 1y ago
  After seeing LangChain abusing the "|" operator overload for pipeline-like DSL, I followed the suite at work and I loved it. It's especially good when you use it in a notebook environment where you literally build the pipeline incrementally through repl.
• true_blue 1y ago
  That new Rhombus language that was featured here recently has an interesting feature where you can use `_` in a function call to act as a "placeholder" for an argument. Essentially it's an easy way to partially apply a function. This works very well with piping because it allows you to pipe into any argument of a function (including optional arguments iirc) rather than just the first like many pipe implementations have. It seems really cool!
  • gdw2 1y ago
    Sounds like Clojure's as-> macro (https://clojuredocs.org/clojure.core/as-%3E https://clojuredocs.org/clojure.core/as-%3E).
• otsukare 1y ago
  I wish more languages would aim for infix functions (like Haskell and Kotlin), rather than specifically the pipe operator.
• Straw 1y ago
  Lisp macros allow a general solution to this that doesn't just handle chained collection operators but allows you to decide the order in which you write any chain of calls. For example, we can write: (foo (bar (baz x))) as (-> x baz bar foo) If there are additional arguments, we can accommodate those too: (sin (* x pi) as (-> x (* pi) sin) Where expression so far gets inserted as the first argument to any form. If you want it inserted as the last argument, you can use ->> instead: (filter positive? (map sin x)) as (->> x (map sin) (filter positive?)) You can also get full control of where to place the previous expression using as->. Full details at https://clojure.org/guides/threading_macros https://clojure.org/guides/threading_macros
  • gleenn 1y ago
    I find the threading operators in Clojure bring much joy and increase readability. I think it's interesting because it makes me actually consider function argument order much more because I want to increase opportunities to use them.
    • aeonik 1y ago
      These threading macros can increase performance, the developer even has a parallelizing threading macro. I use these with xforms transducers. https://github.com/johnmn3/injest https://github.com/johnmn3/injest
  • sooheon 1y ago
    Yes threading macros are so much nicer than method chaining, because it allows general function reuse, rather than being limited to the methods that happen to be defined in your initial data object.
  • benrutter 1y ago
    Yeah, I found this when I was playing around with Hy a while back. I wanted a generic `->` style operator, and isn't wasn't too much trouble to write a macro to introduce one. That's sort of an argument for the existence of macros as a whole, you can't really do this as neatly in something like python (although I've tried) - I can see the downside of working in a codebase with hundreds of these kind of custom language features though.
• wavemode 1y ago
  > At this point you might wonder if Haskell has some sort of pipelining operator, and yes, it turns out that one was added in 2014! That’s pretty late considering that Haskell exists since 1990. The tone of this (and the entire Haskell section of the article, tbh) is rather strange. Operators aren't special syntax and they aren't "added" to the language. Operators are just functions that by default use infix position. (In fact, any function can be called in infix position. And operators can be called in prefix position.) The commit in question added & to the prelude. But if you wanted & (or any other character) to represent pipelining you have always been able to define that yourself. Some people find this horrifying, which is a perfectly valid opinion (though in practice, when working in Haskell it isn't much of a big deal if you aren't foolish with it). But at least get the facts correct.
• relaxing 1y ago
  These articles never explain what’s wrong with calling each function separately and storing each return value in an intermediate variable. Being able to inspect the results of each step right at the point you’ve written it is pretty convenient. It’s readable. And the compiler will optimize it out.
• amai 1y ago
  Pipelining looks nice until you have to debug it. And exception handling is also very difficult, because that means to add forks into your pipelines. Pipelines are only good for programming the happy path.
  • mpalmer 1y ago
    At the risk of over generalized pronouncements, ease of debugging is usually down to how well-designed your tooling happens to be. Most of the time the framework/language does that for you, but it's not the only option. And for exceptions, why not solve it in the data model, and reify failures? Push it further downstream, let your pipeline's nodes handle "monadic" result values. Point being, it's always a tradeoff, but you can usually lessen the pain more than you think. And that's without mentioning that a lot of "pipelining" is pure sugar over the same code we're already writing.
  • bergen 1y ago
    Depends on the context - in a scripting language where you have some kind of console you just don't copy all lines, and see what each pipe does one after another. This is pretty straight forward. (Not talking about compiled code though)
  • bsder 1y ago
    Pipelining is also nice until you have to use it for everything because you can't do alternatives (like default function arguments) properly. Rust chains everything because of this. It's often unpleasant (see: all the Rust GUI toolkits).
  • w4rh4wk5 1y ago
    Yes, certainly! I've encountered and used this pattern in Python, Ruby, Haskell, Rust, C#, and maybe some other languages. It often feels nice to write, but reading can easily become difficult -- especially in Haskell where obscure operators can contain a lot of magic. Debugging them interactively can be equally problematic, depending on the tooling. I'd argue, it's commonly harder to debug a pipeline than the equivalent imperative code and, that in the best case it's equally hard.
  • deleted 1y ago
• dpc_01234 1y ago
  I think there's a language syntax to be invented that would make everything suffix/pipeline-based. Stack based languages are kind of there, but I don't think exactly the same thing. BTW. For people complaining about debug-ability of it: https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.inspect https://doc.rust-lang.org/std/iter/trait.Iterator.html#metho... etc.
  • AndyKluger 1y ago
    What do you think stack based languages like Factor miss in this regard?
    • dpc_01234 1y ago
      I don't want to be manipulating stack. I'd prefer something like Hoon.
• huyegn 1y ago
  I liked the pipelining syntax so much from pyspark and linq that I ended up implementing my own mini linq-like library for python to use in local development. It's mainly used in quick data processing scripts that I run locally. The syntax just makes everything much nicer to work with. https://datapad.readthedocs.io/en/latest/quickstart.html#overview https://datapad.readthedocs.io/en/latest/quickstart.html#ove...
  • michalsustr 1y ago
    Looks really neat, might use that in my work!
• neuroelectron 1y ago
  I really like the website layout. I'm guessing that they're optimizing for Kindle or other e-paper readers.
  • pixelmeister 1y ago
    I recognized this site layout from a past HN post about a solar powered website. Check out their about page. It links to the source for the style that explains why it looks the way it does. Not to spoil it, but it's not for e-readers :)
• okayishdefaults 1y ago
  Surprised that the term "tacit programming" wasn't mentioned once in the article. Point-free style and pipelining were meant for each other. https://en.m.wikipedia.org/wiki/Tacit_programming https://en.m.wikipedia.org/wiki/Tacit_programming
  • Mond_ 1y ago
    Point free was technically mentioned once, but more as a "I'd rather not get into this in great detail right now." thing. It's really cool, though.
• vitus 1y ago
  I think the biggest win for pipelining in SQL is the fact that we no longer have to explain that SQL execution order has nothing to do with query order, and we no longer have to pretend that we're mimicking natural language. (That last point stops being the case when you go beyond "SELECT foo FROM table WHERE bar LIMIT 10".) No longer do we have to explain that expressions are evaluated in the order of FROM -> JOIN -> ON -> SELECT -> WHERE -> GROUP BY -> HAVING -> ORDER BY -> LIMIT (and yes, I know I'm missing several other steps). We can simply just express how our data flows from one statement to the next. (I'm also stating this as someone who has yet to play around with the pipelining syntax, but honestly anything is better than the status quo.)
  • _dark_matter_ 1y ago
    You flipped SELECT and WHERE, which probably just solidifies your point. I can't count the number if times I've seen this trip up analysts.
• ZYbCRq22HbJ2y7 1y ago
  Its nice sugar, but pretty much any modern widely used language supports "pipelining", just not of the SML flavor.
• jongjong 1y ago
  Pipelining is great. Currying is horrible. Though currying superficially looks similar to pipelining. One difference is that currying returns an incomplete result (another function) which must be called again at a later time. On the other hand, pipelining usually returns raw values. Currying returns functions until the last step. The main philosophical failure of currying is that it treats logic/functions as if they were state which should be passed around. This is bad. Components should be responsible for their own state and should just talk to each other to pass plain information. State moves, logic doesn't move. A module shouldn't have awareness of what tools/logic other modules need to do their jobs. This completely breaks the separation of concerns principle. When you call a plumber to fix your drain, do you need to provide them with a toolbox? Do you even need to know what's inside their toolbox? The plumber knows what tools they need. You just show them what the problem is. Passing functions to another module is like giving a plumber a toolbox which you put together by guessing what tools they might need. You're not a plumber, why should you decide what tools the plumber needs? Currying encourages spaghetti code which is difficult to follow when functions are passed between different modules to complete the currying. In practice, if one can design code which gathers all the info it needs before calling the function once; this leads to much cleaner and much more readable code.
• raggi 1y ago
  > (This is not real Rust code. Quick challenge for the curious Rustacean, can you explain why we cannot rewrite the above code like this, even if we import all of the symbols?) Um, you can: #![feature(import_trait_associated_functions)] use Iterator::{collect, map, filter}; fn get_ids2(data: Vec<usize>) -> Vec<usize> { collect(map(filter(<[_]>::iter(&data), |v| ...), |v| ...)) } and you can because it's lazy, which is also the same reason you can write it the other way.. in rust. I think the author was getting at an ownership trap, but that trap is avoided the same way for both arrangements, the call order is the same in both arrangements. If the calls were actually a pipeline (if collect didn't exist and didn't need to be called) then other considerations show up.
  • Mond_ 1y ago
    Guilty as charged, I did not know about the `import_trait_associated_functions` feature at the time. I might add a note to the article to clarify this.
• XorNot 1y ago
  Every example of why this is meant to be good is contrived. You have a create_user function that doesn't error? Has no branches based on type of error? We're having arguments over the best way break these over multiple lines? Like.. why not just store intermediate results in variables? Where our branch logic can just be written inline? And then the flow of data can be very simply determined by reading top to bottom?
• stuaxo 1y ago
  It's part of why JQuery was so great, and the Django ORM.
• invalidator 1y ago
  The author keeps calling it "pipelining", but I think the right term is "method chaining". Compare with a simple pipeline in bash: grep needle < haystack.txt | sed 's/foo/bar/g' | xargs wc -l Each of those components executes in parallel, with the intermediate results streaming between them. You get a similar effect with coroutines. Compare Ruby: data = File.readlines("haystack.txt") .map(&:strip) .grep(/needle/) .map { |i| i.gsub('foo', 'bar') } .map { |i| File.readlines(i).count } In that case, each line is processed sequentially, with a complete array being created between each step. Nothing actually gets pipelined. Despite being clean and readable, I don't tend to do it any more, because it's harder to debug. More often these days, I write things like this: data = File.readlines("haystack.txt") data = data.map(&:strip) data = data.grep(/needle/) data = data.map { |i| i.gsub('foo', 'bar') } data = data.map { |i| File.readlines(i).count } It's ugly, but you know what? I can set a breakpoint anywhere and inspect the intermediate states without having to edit the script in prod. Sometimes ugly and boring is better.
  • wahern 1y ago
    > The author keeps calling it "pipelining", but I think the right term is "method chaining". [...] You get a similar effect with coroutines. The inventor of the shell pipeline, Douglas McIlroy, always understood the equivalency between pipelines and coroutines; it was deliberate. See https://www.cs.dartmouth.edu/~doug/sieve/sieve.pdf https://www.cs.dartmouth.edu/~doug/sieve/sieve.pdf It goes even deeper than it appears, too. The way pipes were originally implemented in the Unix kernel was when the pipe buffer was filled[1] by the writer the kernel continued execution directly in the blocked reader process without bouncing through the scheduler. Effectively, arguably literally, coroutines; one process call the write function and execution continues with a read call returning the data. Interestingly, Solaris Doors operate the same way by design--no bouncing through the scheduler--unlike pipes today where long ago I think most Unix kernels moved away from direct execution switching to better support multiple readers, etc. [1] Or even on the first write? I'd have to double-check the source again.
  • axblount 1y ago
    Syntactic sugar can sometimes fool us into thinking the underlying process is more efficient or streamlined. As a new programmer, I probably would have assumed that "storing" `data` at each step would be more expensive.
• jmyeet 1y ago
  Hack (Facebook's PHP fork) has this feature. It's called pipes [1]: $x = vec[2,1,3] |> Vec\map($$, $a ==> $a * $a) // $$ with value vec[2,1,3] |> Vec\sort($$); // $$ with value vec[4,1,9] It is a nice feature. I do worry about error reporting with any feature that combines multiple statements into a single statement, which is essentially what this does. In Java, there was always an issue with NullPointerExceptiosn being thrown and if you chain several things together you're never sure which one was null. [1]: https://docs.hhvm.com/hack/expressions-and-operators/pipe https://docs.hhvm.com/hack/expressions-and-operators/pipe
  • alganet 1y ago
    Wait. Isn't that already solved in Java? Optional, Mono, Flux, etc. I remember being able to deal with object streams with it quite comfortably.
    • jmyeet 1y ago
      Any function that can return an object in Java can return a null. Nullability not being part of the type system I think is a design fail.
      • alganet 1y ago
        Yeah, so the null checks become annoying if you use only language fundamentals. Java has a culture of having a layer above fundamentals. We're past all that already. I am discussing the ergonomics of their null checking APIs, particularly in the context of pipelining (or streaming, in the Java world). I find them quite comfortable.
• bcoates 1y ago
  Why is the SQL syntax so unnecessarily convoluted? SQL is already an operator language, just an overly constrained one due to historical baggage. If you're going to allow new syntax at all, you can just do from customer left join orders on c_custkey = o_custkey and o_comment not like '%unusual%' group by c_custkey alias count(o_orderkey) as count_of_orders group by count_of_orders alias count(*) as count_of_customers order by count_of_customers desc select count_of_customers, count_of_orders; I'm using 'alias' here as a strawman keyword for what the slide deck calls a free-standing 'as' operator because you can't reuse that keyword, it makes the grammar a mess. The aliases aren't really necessary, you could just write the last line as 'select count(count(*)) ncust, count(*) nord' if you aren't afraid of nested aggregations, and if you are you'll never understand window functions, soo... The |> syntax adds visual noise without expressive power, and the novelty 'aggregate'/'call' operators are weird special-case syntax for something that isn't that complex in the first place. The implicit projection is unnecessary too, for the same reason any decent SQL linter will flag an ambiguous 'select *'
  • zeroimpl 1y ago
    I think they are solving two different problems at the same time. One is the order of elements in a single operation (SELECT then FROM then WHERE etc), and the second is the actual pipelining which replaces the need for nested queries. It does seem like the former could be solved by just loosening up the grammar to allow you to specify things in any order. Eg this seems perfectly unambiguous: from customer group by c_custkey select c_custkey, count(*) as count_of_customers
    • bcoates 1y ago
      Yeah, exactly. You don't need literal pipes
• rokob 1y ago
  I learned the term for this as a fluent interface. Pipelining is in my mind something quite different.
• middayc 1y ago
  Ryelang has a little different take on this ... op-words and pipe-words: https://ryelang.org/meet_rye/specifics/opwords/ https://ryelang.org/meet_rye/specifics/opwords/
• andyferris 1y ago
  Pipelining is great! Though sometimes you want to put the value in the first argument of a function, or a different location, or else call a method... it can be nice to simply refer to the value directly with `_` or `%` or `$` or something. In fact, I always thought it would be a good idea for all statement blocks (in any given programming language) to allow an implicit reference to the value of the previous statement. The pipeline operation would essentially be the existing semicolons (in a C-like language) and there would be a new symbol or keyword used to represent the previous value. For example, the MATLAB REPL allows for referring to the previous value as `ans` and the Julia REPL has inherited the same functionality. You can copy-paste this into the Julia REPL today: [1, 2, 3]; map(x -> x * 2, ans); @show ans; filter(x -> x > 2, ans); @show ans; sum(ans) You can't use this in Julia outside the REPL, and I don't think `ans` is a particularly good keyword for this, but I honestly think the concept is good enough. The same thing in JavaScript using `$` as an example: { [1 ,2, 3]; $.map(x => x * 2); (console.log($), $); $.filter(x => x > 2); (console.log($), $); $.reduce((acc, next) => acc + next, 0) } I feel it would work best with expression-based languages having blocks that return their final value (like Rust) since you can do all sorts of nesting and so-on.
  • cess11 1y ago
    In the Node and Python interpreters you'd use _, in browser JS consoles (and PHP shells like Psysh/Tinker) you'd use $_, in Picolisp @ (or @@ or @@@ for previous computations). I think most interactive programming shells has an equivalent.
• _heimdall 1y ago
  Is pipelining the right term here? I've always used the term "transducer" to describe this kind of process, I picked it up from an episode of FunFunFunction if I'm not mistaken.
  • alganet 1y ago
    Aren't transducers strictly functional terminology? Also, does the name matter if it works the same and has the same properties? Maybe the author called it "pipelines" to avoid functional purists from nitpicking it.
    • _heimdall 1y ago
      Yeah I do think of transducers as a functional paradigm, I read the article as describing a very functional paradigm as well. In the context of a specific programming language feature it seems like terminology would be important, I wasn't trying to nitpick unintentionally.
      • alganet 1y ago
        It is important for the functional guys, and I recognize the importance it has for them. These "pipelines" and "object streaming" APIs are often built upon OOP. I feel that calling it "transducers" would offend the sensibilities of those who think it must be functional all the way down. Don't you think it's better to keep it with a different name? I mean, even among the functional community itself there seems to be a lot of stress around purity, why would anyone want to make it worse?
        • _heimdall 1y ago
          I may have just misunderstood the OP. It sounded to me like describing the benefits specifically of transducers, but if it was OOP and more just about piping operators or chaining the term wouldn't fit.
          • 1y ago
• EnPissant 1y ago
  I don't know. I find this: fn get_ids(data: Vec<Widget>) -> Vec<Id> { let mut result = Vec::new(); for widget in &data { if widget.alive { result.push(widget.id); } } result } more readable than this: fn get_ids(data: Vec<Widget>) -> Vec<Id> { data.iter() .filter(|w| w.alive) .map(|w| w.id) .collect() } and I also dislike Rust requiring you to write "mut" for function mutable values. It's mostly just busywork and dogma.
  • Mond_ 1y ago
    Yeah, I really wanted to avoid a discussion over functional vs. imperative programming, so I just... didn't talk about the imperative style at all, and just said so in the first section. I think the imperative style isn't as readable (of course I would), but that's absolutely a discussion for another day, and I get why people prefer it.
    • EnPissant 1y ago
      I think it’s important to point out what the imperative version would look like because I think the fundamental reason that the method chaining approach is more readable is because it more closely resembles imperative code. When reading it, you start with a vector and then you mutate it in various ways before returning it. I understand that’s not how it’s implemented under the hood, but I don’t think it really matters.
• jesse__ 1y ago
  I've always wondered why more languages don't do this. It just makes sense
• jiggawatts 1y ago
  PowerShell has the best pipeline capability of any language I have ever seen. For comparison, UNIX pipes support only trivial byte streams from output to input. PowerShell allows typed object streams where the properties of the object are automatically wired up to named parameters of the commands on the pipeline. Outputs at any stage can not only be wired directly to the next stage but also captured into named variables for use later in the pipeline. Every command in the pipeline also gets begin/end/cancel handlers automatically invoked so you can set up accumulators, authentication, or whatever. UNIX scripting advocates don’t know what they’re missing out on…
  • account-5 1y ago
    You should look at Nushell. Much as I like powershell, Nushell just seems better.
    • jiggawatts 1y ago
      I’d love to, but even the core concepts like pipeline behaviour aren’t documented. There’s just a bunch of empty headings: https://www.nushell.sh/lang-guide/chapters/pipelines.html https://www.nushell.sh/lang-guide/chapters/pipelines.html
      • account-5 1y ago
        Yeah, it's younger than powershell so there's some rough edges in some places. I wouldn't let empty headings in the docs stop you using/experimenting with it though. I use powershell daily but am hopeful that I can replace it with Nushell at some point.
  • replwoacause 1y ago
    Completely agree. PowerShell is what got me hooked on scripting/programming and the pipeline is such an elegant construct.
• rocqua 1y ago
  The left associativity of functions really doesn't work well with English reading left to right. I found this especially clear with the'composition opperator' of functions. Where f.g has to mean f _after_ g because you really want: f.g = f(g(x)) Based on this, I think a reverse polish type of notation would be a lot better. Though perhaps it is a lot nicer to think of "the sine of an angle" than "angle sine-ed". Not that it matters much, the switching costs are immense. Getting people able to teach it would be impossible, and collaboration with people taught in the other system would be horrible. I am doubtful I could make the switch, even if I wanted.
• Weryj 1y ago
  LINQ was my gateway drug into functional programming, Pipelining is so beautiful.
• bjourne 1y ago
  In concatenative languages with an implicit stack (Factor) that expression would read: iter [ alive? ] filter [ id>> ] map collect The beauty of this is that everything can be evaluated strictly left-to-right. Every single symbol. "Pipelines" in other languages are never fully left-to-right evaluated. For example, ".filter(|w| w.alive)" in the author's example requires one to switch from postfix to infix evaluation to evaluate the filter application. The major advantage is that handling multiple streams is natural. Suppose you want to compute the dot product of two files where each line contains a float: fileA fileB [ lines [ str>float ] map ] bi@ [ mul ] 2map 0 [ + ] reduce
  • AndyKluger 1y ago
    Thank you! Because I love to practice and demonstrate Factor, this is working code for that example: "f1.txt" "f2.txt" [ utf8 file-lines [ string>number ] map ] bi@ vdot
• kissgyorgy 1y ago
  In Nix, you can do something like this: gitRef = with lib; pipe .git/HEAD [ readFile trim (splitString ":") last trim (ref: ./git/${ref}) readFile trim ]; Super clean and cool!
• davemp 1y ago
  Computer scientists continue to pick terrible names. Pipelining is already an overloaded concept that implies some type of operation level parallelism. Picking names like this does everyone in the field a disservice. Calling it something like “composition chain” would be much clearer with respect to existing literature in the field. Maybe I’m being nitpicky, but sometimes it feels like the tower of babel parable talking to folks who use different ecosystems.
• immibis 1y ago
  We had this - it was called variables. You could do: x = iter(data); y = filter(x, w=>w.isAlive); z = map(y, w=>w.id); return collect(z); It doesn't need new syntax, but to implement this with the existing syntax you do have to figure out what the intermediate objects are, but you also have that problem with "pipelining" unless it compiles the whole chain into a single thing a la Linq.
• moralestapia 1y ago
  (Un)surprisingly, the author ignores this is almost already a thing in JS. What a terrible oversight. Anyway, JS wins again, give it a try if you haven't, it's one of the best languages out there.
  • jplusequalt 1y ago
    TypeScript is a beauty to work with. It's a shame that it's most often used in web contexts though. I would love to write a game engine that uses TypeScript as the scripting language.
    • moralestapia 1y ago
      Indeed!
• HackerThemAll 1y ago
  I like how they are unaware of neat F# pipelines or else how they deliberately "forgot" to mention them.
• snthpy 1y ago
  Nice post and I very much agree! In fact I tried to make some similar points in my CMU "SQL or Death" Seminar Series talk on PRQL (https://db.cs.cmu.edu/events/sql-death-prql-pipelined-relational-query-language/ https://db.cs.cmu.edu/events/sql-death-prql-pipelined-relati...) in that I would love to see PRQL (or something like it) become a universal DSL for data pipelines. Ideally this wouldn't even have to go through some query engine and could just do some (byte)codegen for your target language. P.S. Since you mentioned the Google Pipe Syntax HYTRADBOI 2025 talk, I just want to throw out that I also have a 10 min version for the impatient: https://www.hytradboi.com/2025/deafce13-67ac-40fd-ac4b-175d53318a78-prql-a-modern-pipelined-sql-replacement https://www.hytradboi.com/2025/deafce13-67ac-40fd-ac4b-175d5... That's just a PRQL overview though. The Universal Data Pipeline DSL ideas and comparison to LINQ, F#, ... are only in the CMU talk. I also go a bit into imperative vs declarative and point out that since "pipelining" is just function composition it should really be "functional" rather than imperative or declarative (which also came up in this thread).