4 ms·

Elixir-style pipelines in Ruby

63 points by gregnavis 4y ago

ilyash 4y ago
While not as thorough solution as mentioned here in comments, UFCS goes a long way in this direction. In Next Generation Shell, I've designed the methods so that the first parameter is something that is likely to come from "the pipeline". Hence mylist.filter(...).map(...) just work. Combined with multiple dispatch and the fact that methods don't belong to a particular type/class, it allows creating user-defined methods with same convention to work with any existing and new types/classes. UFCS - https://en.m.wikipedia.org/wiki/Uniform_Function_Call_Syntax https://en.m.wikipedia.org/wiki/Uniform_Function_Call_Syntax
bwilliams 4y ago
The beauty (and horror) of Ruby is that you can do almost anything with it. I think this is a really interesting and clever use of the "can do anything" aspect of Ruby, although I think I'd prefer not to run into it in a production app. Still, it's really cool to see how far we can push/mold the language to accomplish different tasks and patterns.
zelphirkalt 4y ago
Do the steps of the pipeline implementation in Ruby run concurrently? I once did an Elixir course and really liked the pipelines. I continued implementing pipelines with a Scheme macro, but not concurrently.
- Fire-Dragon-DoL 4y ago
  Elixir pipelines are not concurrent, unless you use something that allows that (flow and probably stream behind the scenes)
- woodruffw 4y ago
  Not naively: this kind of pipelining is partial function application, not coordinated parallelism (the way Unix pipeliens are).
  - AtlasBarfed 4y ago
    Does anyone know any documentation on how UNIX achieves parallelism in pipelines, especially modern implementations? Is each stage of a piped command just another spun off process?
    - woodruffw 4y ago
      Yes, it's just that: each process in the pipeline is spawned, and then blocks as it waits for input from the previous step in the pipeline. I think a better way to think about it is that the UNIX model requires parallelism: without parallelism, each stage in the pipeline would need to fully buffer its intermediate state before forwarding it to the next stage.
      - superdisk 4y ago
        > without parallelism, each stage in the pipeline would need to fully buffer its intermediate state before forwarding it to the next stage. That's how DOS implemented the pipe syntax I believe.
    - masklinn 4y ago
      > Is each stage of a piped command just another spun off process? Yes. With, as the name says, a pipe inbetween.
bonquesha99 4y ago
Check out this proof of concept gem to perform pipe operations in Ruby using block expressions: https://github.com/lendinghome/pipe_operator#-pipe_operator https://github.com/lendinghome/pipe_operator#-pipe_operator "https://api.github.com/repos/ruby/ruby".pipe do URI.parse Net::HTTP.get JSON.parse.fetch("stargazers_count") yield_self { |n| "Ruby has #{n} stars" } Kernel.puts end #=> Ruby has 15120 stars
quechimba 4y ago
I like using .then for chaining stuff, like this: sig { params(obj: T.untyped).void } def write(obj) obj .then { Array(_1) } .then { @wrapper.pack(_1) } .then { @deflate.deflate(_1, Zlib::SYNC_FLUSH) } .then { @body.write(_1) } end
joeman1000 4y ago
I've had a look at threading/piping operators in a few languages (list below). I'd say that Racket has the best one I've used. I love that you can specify a '_' for the hole which the result from the previous operation will fill. Julia's threading macro is surprisingly brittle, only letting you chain single-argument functions unless you want to use anonymous functions with one bound variable and the rest free. + Haskell :: https://www.schoolofhaskell.com/user/Gabriel439/Pipes%20tutorial https://www.schoolofhaskell.com/user/Gabriel439/Pipes%20tuto... + Racket :: https://docs.racket-lang.org/threading/index.html https://docs.racket-lang.org/threading/index.html + Clojure :: https://clojure.org/guides/threading_macros https://clojure.org/guides/threading_macros + Julia :: https://syl1.gitbook.io/julia-language-a-concise-tutorial/useful-packages/pipe https://syl1.gitbook.io/julia-language-a-concise-tutorial/us... + R :: https://r4ds.had.co.nz/pipes.html https://r4ds.had.co.nz/pipes.html
- sundarurfriend 4y ago
  > Julia's threading macro is surprisingly brittle, only letting you chain single-argument functions Julia's threading/piping is surprisingly limited, but the phrasing here (combined with the link) could make things confusing to a Julia beginner. So to make things clear: * The linked page is talking about an external Julia package which provides a macro. And that macro lets you use the `_` syntax similar to what you describe Racket as having. * Julia's default inbuilt piping syntax is the one with the single-argument limitation, and that's an operator, not a macro. There's been a lot of discussion about bringing the `_` syntax or something like it to the base language, but there seem to be implementation difficulties. [1] > +R :: https://r4ds.had.co.nz/pipes.html https://r4ds.had.co.nz/pipes.html This page is talking about magrittr piping (which is probably still the most popular), but base R also got inbuilt piping syntax with version 4.1. It automatically passes the piped-in value as the first argument to the subsequent function. [2] [3] [1] https://github.com/JuliaLang/julia/pull/24990 https://github.com/JuliaLang/julia/pull/24990 [2] https://www.r-bloggers.com/2021/05/the-new-r-pipe/ https://www.r-bloggers.com/2021/05/the-new-r-pipe/ [3] https://michaelbarrowman.co.uk/post/the-new-base-pipe/ https://michaelbarrowman.co.uk/post/the-new-base-pipe/
- fulafel 4y ago
  > I'd say that Racket has the best one I've used. I love that you can specify a '_' for the hole which the result from the previous operation will fill. Clojure also has a placeholder variant, the as-> form. Quoting the example the Clojure doc page (https://clojure.org/guides/threading_macros https://clojure.org/guides/threading_macros): (as-> [:foo :bar] v (map name v) (first v) (.substring v 1)) ;; => "oo"

63 points by gregnavis 4y ago

While not as thorough solution as mentioned here in comments, UFCS goes a long way in this direction. In Next Generation Shell, I've designed the methods so that the first parameter is something that is likely to come from "the pipeline". Hence mylist.filter(...).map(...) just work. Combined with multiple dispatch and the fact that methods don't belong to a particular type/class, it allows creating user-defined methods with same convention to work with any existing and new types/classes. UFCS - https://en.m.wikipedia.org/wiki/Uniform_Function_Call_Syntax https://en.m.wikipedia.org/wiki/Uniform_Function_Call_Syntax

bwilliams 4y ago

The beauty (and horror) of Ruby is that you can do almost anything with it. I think this is a really interesting and clever use of the "can do anything" aspect of Ruby, although I think I'd prefer not to run into it in a production app. Still, it's really cool to see how far we can push/mold the language to accomplish different tasks and patterns.

zelphirkalt 4y ago

Do the steps of the pipeline implementation in Ruby run concurrently? I once did an Elixir course and really liked the pipelines. I continued implementing pipelines with a Scheme macro, but not concurrently.

Fire-Dragon-DoL 4y ago
Elixir pipelines are not concurrent, unless you use something that allows that (flow and probably stream behind the scenes)
woodruffw 4y ago
Not naively: this kind of pipelining is partial function application, not coordinated parallelism (the way Unix pipeliens are).
- AtlasBarfed 4y ago
  Does anyone know any documentation on how UNIX achieves parallelism in pipelines, especially modern implementations? Is each stage of a piped command just another spun off process?
  - woodruffw 4y ago
    Yes, it's just that: each process in the pipeline is spawned, and then blocks as it waits for input from the previous step in the pipeline. I think a better way to think about it is that the UNIX model requires parallelism: without parallelism, each stage in the pipeline would need to fully buffer its intermediate state before forwarding it to the next stage.
    - superdisk 4y ago
      > without parallelism, each stage in the pipeline would need to fully buffer its intermediate state before forwarding it to the next stage. That's how DOS implemented the pipe syntax I believe.
  - masklinn 4y ago
    > Is each stage of a piped command just another spun off process? Yes. With, as the name says, a pipe inbetween.

bonquesha99 4y ago

Check out this proof of concept gem to perform pipe operations in Ruby using block expressions: https://github.com/lendinghome/pipe_operator#-pipe_operator https://github.com/lendinghome/pipe_operator#-pipe_operator "https://api.github.com/repos/ruby/ruby".pipe do URI.parse Net::HTTP.get JSON.parse.fetch("stargazers_count") yield_self { |n| "Ruby has #{n} stars" } Kernel.puts end #=> Ruby has 15120 stars

quechimba 4y ago

I like using .then for chaining stuff, like this: sig { params(obj: T.untyped).void } def write(obj) obj .then { Array(_1) } .then { @wrapper.pack(_1) } .then { @deflate.deflate(_1, Zlib::SYNC_FLUSH) } .then { @body.write(_1) } end

joeman1000 4y ago

I've had a look at threading/piping operators in a few languages (list below). I'd say that Racket has the best one I've used. I love that you can specify a '_' for the hole which the result from the previous operation will fill. Julia's threading macro is surprisingly brittle, only letting you chain single-argument functions unless you want to use anonymous functions with one bound variable and the rest free. + Haskell :: https://www.schoolofhaskell.com/user/Gabriel439/Pipes%20tutorial https://www.schoolofhaskell.com/user/Gabriel439/Pipes%20tuto... + Racket :: https://docs.racket-lang.org/threading/index.html https://docs.racket-lang.org/threading/index.html + Clojure :: https://clojure.org/guides/threading_macros https://clojure.org/guides/threading_macros + Julia :: https://syl1.gitbook.io/julia-language-a-concise-tutorial/useful-packages/pipe https://syl1.gitbook.io/julia-language-a-concise-tutorial/us... + R :: https://r4ds.had.co.nz/pipes.html https://r4ds.had.co.nz/pipes.html

sundarurfriend 4y ago
> Julia's threading macro is surprisingly brittle, only letting you chain single-argument functions Julia's threading/piping is surprisingly limited, but the phrasing here (combined with the link) could make things confusing to a Julia beginner. So to make things clear: * The linked page is talking about an external Julia package which provides a macro. And that macro lets you use the `_` syntax similar to what you describe Racket as having. * Julia's default inbuilt piping syntax is the one with the single-argument limitation, and that's an operator, not a macro. There's been a lot of discussion about bringing the `_` syntax or something like it to the base language, but there seem to be implementation difficulties. [1] > +R :: https://r4ds.had.co.nz/pipes.html https://r4ds.had.co.nz/pipes.html This page is talking about magrittr piping (which is probably still the most popular), but base R also got inbuilt piping syntax with version 4.1. It automatically passes the piped-in value as the first argument to the subsequent function. [2] [3] [1] https://github.com/JuliaLang/julia/pull/24990 https://github.com/JuliaLang/julia/pull/24990 [2] https://www.r-bloggers.com/2021/05/the-new-r-pipe/ https://www.r-bloggers.com/2021/05/the-new-r-pipe/ [3] https://michaelbarrowman.co.uk/post/the-new-base-pipe/ https://michaelbarrowman.co.uk/post/the-new-base-pipe/
fulafel 4y ago
> I'd say that Racket has the best one I've used. I love that you can specify a '_' for the hole which the result from the previous operation will fill. Clojure also has a placeholder variant, the as-> form. Quoting the example the Clojure doc page (https://clojure.org/guides/threading_macros https://clojure.org/guides/threading_macros): (as-> [:foo :bar] v (map name v) (first v) (.substring v 1)) ;; => "oo"