5 ms·

Pipeline-Oriented Programming [video]

61 points by ingve 3y ago

deleted 3y ago
[deleted]
tandav 3y ago
shameless plug: I maintain a small library to do functional pipes. You can write: ( range(10) | Map(lambda x: x * 10) | Filter(lambda x: x % 2 == 0) | Reduce(lambda a, b: a + b) ) and more. https://tandav.github.io/pipe21/ https://tandav.github.io/pipe21/ most operators are basically oneliners: https://github.com/tandav/pipe21/blob/master/pipe21.py#L17 https://github.com/tandav/pipe21/blob/master/pipe21.py#L17 I also maintain a list of libraries which uses pipeline/chain syntax: https://tandav.github.io/pipe21/similar-tools/ https://tandav.github.io/pipe21/similar-tools/
- throwaway914 3y ago
  This is very cool. I remember I got sucked into things like Ramda going down this functional programming rabbit hole :-) https://ramdajs.com/ https://ramdajs.com/
  - retrocryptid 3y ago
    The first thing I did when I encountered js in the 90s was write map and reduce. After leaving Mozilla I published a npm package with my hacks. I was very happy to find someone else had done a much better job at it and was actually maintaining it. also, have you seen lazyjs? Also very cool.
scns 3y ago
Slides: https://fsharpforfunandprofit.com/pipeline/ https://fsharpforfunandprofit.com/pipeline/ Code: https://github.com/swlaschin/pipeline_oriented_programming_talk https://github.com/swlaschin/pipeline_oriented_programming_t...
- akkad33 3y ago
  This seems to be a rare update to the site which hasn't seen new content in more than 3 years?
bob1029 3y ago
Expressions (LINQ) are the most powerful feature available in C#. There isn't much you can't do with them if you get a bit clever. I can't imagine how many total hours of development time these abstractions have saved. I also find it relatively easy to explain these bits of code to non-developers. Performance is a common argument for not utilizing code like this, but modern .NET does all kinds of insane runtime optimization techniques to make sure things go fast (if they need to!). We just recently started looking at using EFCore (instead of raw SQL via Dapper) and it is dawning on me that it was a strategic mistake to not further utilize the leverage that expressions can bring to the party.
- louthy 3y ago
  If you want to see this taken to the max, then my library [1] has many monadic types (which is what LINQ is for) implemented. Including an implementation of Haskell Pipes [2], with some examples [3] [1] https://github.com/louthy/language-ext https://github.com/louthy/language-ext [2] https://louthy.github.io/language-ext/LanguageExt.Core/Effects/Pipes/index.html https://louthy.github.io/language-ext/LanguageExt.Core/Effec... [3] https://github.com/louthy/language-ext/blob/main/Samples/EffectsExamples/Examples/TextFileChunkStreamExample.cs https://github.com/louthy/language-ext/blob/main/Samples/Eff...
  - lacrimacida 3y ago
    For some reason your github link repeatedly crashes for me. Not sure whether it’s because im trying to view it from my iphone..
    - bob1029 3y ago
      Crashes for me too on Safari/M1 MBP. Github's client-side rendering "enhancements" strike again. Maybe I'll submit another cranky support ticket this weekend...
      - zdragnar 3y ago
        Android Chrome loads the page but scrolling is rather janky, so i suspect you may well be right.
AndrewDucker 3y ago
This is also how PowerShell works. It's incredibly effective for generating/fetching data and processing it.
- 6c696e7578 3y ago
  Unless things have changed the pipelines in powershell are collectors, unlike fifos in unix. For small tasks it's fine, but when shifting large blocks of data it can be noticeable slower.
  - AndrewDucker 3y ago
    The pipelines pass one item at a time unless you deliberately make them into collectors.
    - 6c696e7578 3y ago
      From memory I was doing something like write-output $var within a loop. My assumption was that a single item was readable at write time by the right hand side of the pipe. It built up something like a gig in memory before it was readable, which wasn't what I was expecting. What would have been the way to make that more like a unix pipe? I didn't get to the bottom of it as time didn't permit.
louthy 3y ago
Not sure why Scott Wlaschin needs to keep renaming existing concepts ('Monads' to 'Railway Oriented Programming' and 'Function Composition' to 'Pipeline Oriented Programming'), but I really don't think it helps shared understanding. It feels, at worst, like the height of arrogance to assign new terms to existing concepts, as though you've just invented the concept. And, at best, a form of anti-intellectualism - to try and make functional concepts accessible to ex-OO devs. But it feels to me like all it does is muddy the waters. Not everything needs to be oriented programming.
- jmaker 3y ago
  It might be purely marketing-oriented, or perhaps it’s just sales-funnel-oriented, who knows. And if anything unexpected pops up, look no further, the author can immediately come up with a new name for a decades old solution. Same happened to the “hexagonal,” “ports-and-adapters” and so on, which I noted mostly in the .NET community btw, correct me if that's wrong. Like you know every new consulting agency coming up with some new names for old stuff to stand out perhaps?
- eterps 3y ago
  For what it's worth, it helped me a great deal in understanding.
  - AnimalMuppet 3y ago
    Yeah. If the new term conveys the actual meaning better than the old term, then why not invent a new term when dealing with the new people? We didn't always fully understand what X was when we coined X as the term for it; sometimes better names are possible. Why not start to use them? Or, probably more often, a new term for the same old thing is not better, serves no useful purpose, and just confuses.
    - retrocryptid 3y ago
      Because The lisp community has been doing this for 50 years and has a different name for it.
jbverschoor 3y ago
Strange that Elixir pipes are not mentioned at all..
- fredrikholm 3y ago
  Why? The talk is about F#.
  - retrocryptid 3y ago
    There's a community that has experience doing analysis, design and coding with functional composition as an abnstraction. It's useful to reference that community so the viewer can get more information and more examples on applying it. Also, sooner or later people will notice difficulties in communicating exceptional events up and down the chain and they would be greatly served by how Erlang/Elixir handles errors and process trees. Even if you want to stay in c# or f#, there's still wisdom to be gained by looking at how things work in functional languages and applying what you can in non functional languages.
    - fredrikholm 3y ago
      Sure, but the talk is in a very .NET heavy conference filled with C#-ers, talking about F#, from a guy whose entire internet persona is advocating F#. It's not strange that he doesn't mention SML, Caml, OCaml, Miranda, Haskell etc given this context.
bjourne 3y ago
Pipelines are effective for expressing functional concepts but are not suitable for general programming. E.g writing merge sort or a chaining hash table as a pipeline won't work out well. The problem is state. There are dataflow programming languages like Lustre and Esterel that tried to incorporate state as back arrows in their pipelines. It didn't work out so well.
- anonzzzies 3y ago
  But most things most people do during the day is not general programming. it would benefit from having to use principles like this instead of using 'general programming' to do them in a way that's harder to understand, harder to debug etc.
  - AnimalMuppet 3y ago
    > But most things most people do during the day is not general programming. Well, in one sense, nobody ever writes a general program - they only write specific ones. But if your claim is "most things most people do during the day could be pipelined", I'd like to see your supporting evidence, because I'm not at all sure that claim is true. For example, as far as I understand, GUIs are very hard to pipeline. How much of the programming done in the average day is on GUIs?
    - anonzzzies 3y ago
      > I'd like to see your supporting evidence, because I'm not at all sure that claim is true I believe it is, but of course it's hard to prove; I have my own libs in whatever language I use which are piping libraries very much inspired by FP, LINQ, Haskell, APL/k and other such things. And most things I do, do fit. But yeah; how to show that; I can translate most things we write anyway. We write SaaS in the broadest sense. > For example, as far as I understand, GUIs are very hard to pipeline. Are they? Aren't we pulling data in reducers, transforming it to structures that can be shown in the templates and transforming again on input/click and sending somewhere? It's all pipes, kind of. Sure, some things are not, but those things at least we don't encounter very often. And when they are pipes, other ways of processing are just plainly worse in every way imho.
AndyKluger 3y ago
If anyone's interested to see the slide examples but in Factor, I wrote them out: https://programming.dev/comment/6718327 https://programming.dev/comment/6718327 Here's a bit: : add1 ( x -- x' ) 1 + ; : square ( x -- x' ) dup * ; : double ( x -- x' ) 2 * ; : demo1 ( -- n ) 5 add1 square double ;

61 points by ingve 3y ago

deleted 3y ago

[deleted]

shameless plug: I maintain a small library to do functional pipes. You can write: ( range(10) | Map(lambda x: x * 10) | Filter(lambda x: x % 2 == 0) | Reduce(lambda a, b: a + b) ) and more. https://tandav.github.io/pipe21/ https://tandav.github.io/pipe21/ most operators are basically oneliners: https://github.com/tandav/pipe21/blob/master/pipe21.py#L17 https://github.com/tandav/pipe21/blob/master/pipe21.py#L17 I also maintain a list of libraries which uses pipeline/chain syntax: https://tandav.github.io/pipe21/similar-tools/ https://tandav.github.io/pipe21/similar-tools/

throwaway914 3y ago
This is very cool. I remember I got sucked into things like Ramda going down this functional programming rabbit hole :-) https://ramdajs.com/ https://ramdajs.com/
- retrocryptid 3y ago
  The first thing I did when I encountered js in the 90s was write map and reduce. After leaving Mozilla I published a npm package with my hacks. I was very happy to find someone else had done a much better job at it and was actually maintaining it. also, have you seen lazyjs? Also very cool.

scns 3y ago

Slides: https://fsharpforfunandprofit.com/pipeline/ https://fsharpforfunandprofit.com/pipeline/ Code: https://github.com/swlaschin/pipeline_oriented_programming_talk https://github.com/swlaschin/pipeline_oriented_programming_t...

akkad33 3y ago
This seems to be a rare update to the site which hasn't seen new content in more than 3 years?

bob1029 3y ago

Expressions (LINQ) are the most powerful feature available in C#. There isn't much you can't do with them if you get a bit clever. I can't imagine how many total hours of development time these abstractions have saved. I also find it relatively easy to explain these bits of code to non-developers. Performance is a common argument for not utilizing code like this, but modern .NET does all kinds of insane runtime optimization techniques to make sure things go fast (if they need to!). We just recently started looking at using EFCore (instead of raw SQL via Dapper) and it is dawning on me that it was a strategic mistake to not further utilize the leverage that expressions can bring to the party.

louthy 3y ago
If you want to see this taken to the max, then my library [1] has many monadic types (which is what LINQ is for) implemented. Including an implementation of Haskell Pipes [2], with some examples [3] [1] https://github.com/louthy/language-ext https://github.com/louthy/language-ext [2] https://louthy.github.io/language-ext/LanguageExt.Core/Effects/Pipes/index.html https://louthy.github.io/language-ext/LanguageExt.Core/Effec... [3] https://github.com/louthy/language-ext/blob/main/Samples/EffectsExamples/Examples/TextFileChunkStreamExample.cs https://github.com/louthy/language-ext/blob/main/Samples/Eff...
- lacrimacida 3y ago
  For some reason your github link repeatedly crashes for me. Not sure whether it’s because im trying to view it from my iphone..
  - bob1029 3y ago
    Crashes for me too on Safari/M1 MBP. Github's client-side rendering "enhancements" strike again. Maybe I'll submit another cranky support ticket this weekend...
    - zdragnar 3y ago
      Android Chrome loads the page but scrolling is rather janky, so i suspect you may well be right.

AndrewDucker 3y ago

This is also how PowerShell works. It's incredibly effective for generating/fetching data and processing it.

6c696e7578 3y ago
Unless things have changed the pipelines in powershell are collectors, unlike fifos in unix. For small tasks it's fine, but when shifting large blocks of data it can be noticeable slower.
- AndrewDucker 3y ago
  The pipelines pass one item at a time unless you deliberately make them into collectors.
  - 6c696e7578 3y ago
    From memory I was doing something like write-output $var within a loop. My assumption was that a single item was readable at write time by the right hand side of the pipe. It built up something like a gig in memory before it was readable, which wasn't what I was expecting. What would have been the way to make that more like a unix pipe? I didn't get to the bottom of it as time didn't permit.

louthy 3y ago

Not sure why Scott Wlaschin needs to keep renaming existing concepts ('Monads' to 'Railway Oriented Programming' and 'Function Composition' to 'Pipeline Oriented Programming'), but I really don't think it helps shared understanding. It feels, at worst, like the height of arrogance to assign new terms to existing concepts, as though you've just invented the concept. And, at best, a form of anti-intellectualism - to try and make functional concepts accessible to ex-OO devs. But it feels to me like all it does is muddy the waters. Not everything needs to be oriented programming.

jmaker 3y ago
It might be purely marketing-oriented, or perhaps it’s just sales-funnel-oriented, who knows. And if anything unexpected pops up, look no further, the author can immediately come up with a new name for a decades old solution. Same happened to the “hexagonal,” “ports-and-adapters” and so on, which I noted mostly in the .NET community btw, correct me if that's wrong. Like you know every new consulting agency coming up with some new names for old stuff to stand out perhaps?
eterps 3y ago
For what it's worth, it helped me a great deal in understanding.
- AnimalMuppet 3y ago
  Yeah. If the new term conveys the actual meaning better than the old term, then why not invent a new term when dealing with the new people? We didn't always fully understand what X was when we coined X as the term for it; sometimes better names are possible. Why not start to use them? Or, probably more often, a new term for the same old thing is not better, serves no useful purpose, and just confuses.
  - retrocryptid 3y ago
    Because The lisp community has been doing this for 50 years and has a different name for it.

jbverschoor 3y ago

Strange that Elixir pipes are not mentioned at all..

fredrikholm 3y ago
Why? The talk is about F#.
- retrocryptid 3y ago
  There's a community that has experience doing analysis, design and coding with functional composition as an abnstraction. It's useful to reference that community so the viewer can get more information and more examples on applying it. Also, sooner or later people will notice difficulties in communicating exceptional events up and down the chain and they would be greatly served by how Erlang/Elixir handles errors and process trees. Even if you want to stay in c# or f#, there's still wisdom to be gained by looking at how things work in functional languages and applying what you can in non functional languages.
  - fredrikholm 3y ago
    Sure, but the talk is in a very .NET heavy conference filled with C#-ers, talking about F#, from a guy whose entire internet persona is advocating F#. It's not strange that he doesn't mention SML, Caml, OCaml, Miranda, Haskell etc given this context.

bjourne 3y ago

Pipelines are effective for expressing functional concepts but are not suitable for general programming. E.g writing merge sort or a chaining hash table as a pipeline won't work out well. The problem is state. There are dataflow programming languages like Lustre and Esterel that tried to incorporate state as back arrows in their pipelines. It didn't work out so well.

anonzzzies 3y ago
But most things most people do during the day is not general programming. it would benefit from having to use principles like this instead of using 'general programming' to do them in a way that's harder to understand, harder to debug etc.
- AnimalMuppet 3y ago
  > But most things most people do during the day is not general programming. Well, in one sense, nobody ever writes a general program - they only write specific ones. But if your claim is "most things most people do during the day could be pipelined", I'd like to see your supporting evidence, because I'm not at all sure that claim is true. For example, as far as I understand, GUIs are very hard to pipeline. How much of the programming done in the average day is on GUIs?
  - anonzzzies 3y ago
    > I'd like to see your supporting evidence, because I'm not at all sure that claim is true I believe it is, but of course it's hard to prove; I have my own libs in whatever language I use which are piping libraries very much inspired by FP, LINQ, Haskell, APL/k and other such things. And most things I do, do fit. But yeah; how to show that; I can translate most things we write anyway. We write SaaS in the broadest sense. > For example, as far as I understand, GUIs are very hard to pipeline. Are they? Aren't we pulling data in reducers, transforming it to structures that can be shown in the templates and transforming again on input/click and sending somewhere? It's all pipes, kind of. Sure, some things are not, but those things at least we don't encounter very often. And when they are pipes, other ways of processing are just plainly worse in every way imho.

AndyKluger 3y ago

If anyone's interested to see the slide examples but in Factor, I wrote them out: https://programming.dev/comment/6718327 https://programming.dev/comment/6718327 Here's a bit: : add1 ( x -- x' ) 1 + ; : square ( x -- x' ) dup * ; : double ( x -- x' ) 2 * ; : demo1 ( -- n ) 5 add1 square double ;