5 ms·
Composable Pipelines [in Go] Improved
- ceocoder 11y agoHey 'samuell, I think you meant, func NewProcess() *A-Process { ....
- hden 11y agoJust use a transducer. https://www.youtube.com/watch?v=6mTbuzafcII https://www.youtube.com/watch?v=6mTbuzafcII
- samuell 11y agoWill have a look!
- secure 11y agoThe example code doesn’t actually compile: https://play.golang.org/p/Fv-Sf1OJdH https://play.golang.org/p/Fv-Sf1OJdH I think it should read: https://play.golang.org/p/vrwEkHy-DR https://play.golang.org/p/vrwEkHy-DR
- samuell 11y agoThanks! I have pushed a lot of fixes now and sent a PR. In the meanwhile, please see this working example on the Go playground: http://play.golang.org/p/voUfPGQulf http://play.golang.org/p/voUfPGQulf
- hb42 11y agoWithout any parametric polymorphism, this kind of utility functions are hardly composable at all. At best these are copy-pastable patterns.
- samuell 11y agoThis sounds like an interesting observation, but could you please elaborate a bit more?
- catnaroek 11y agoWith parametric polymorphism and generalized algebraic data types, you could: (0) Give every process in the pipeline could its own input and output type, rather than forcing all processes to use strings. (1) Enforce, statically, that only "compatible" processes can be composed. For instance, if `Foo` is a process that outputs integers, and `Bar` is another process that takes strings as input, then `Foo` can't feed its output to `Bar`. However, this is involves some rather tricky type hackery (so-called "type-aligned sequences"). Hardly a good fit for a language with Go's design goals.
- samuell 11y agoOk, I think I see what you mean. Well, I think unfortunately my sketchy code examples maybe did not cover the idea in enough detail. What I'm doing in practice is to create different struct types for each type of process I use, which all are of a "base interface" called "process" (mainly for the Run() method, so that the pipeline component can call it). An example of this in action is shown in this code example on the Go playground: http://play.golang.org/p/voUfPGQulf http://play.golang.org/p/voUfPGQulf The typing of channels for each process struct type should now enforce what components can be connected together (you will not be able to assign a "string chan" to a "[]byte chan" field, for example). Did this answer the question, or do you see further problems here?
- catnaroek 11y agoAh, okay. I misunderstood the intention of your original example, then. My bad. In order to prevent further embarrassing myself, I'm going to tell you how I (currently) understand your design, and, if I'm wrong, please tell me: Your "pipeline" isn't actually connecting the processes to each other. You first connect the processes to each other, and only then add them to a "pipeline", whose primary (only?) responsibility is to ensure that each process runs in a different goroutine. This much doesn't really need parametric polymorphism, let alone generalized algebraic data types. However, calling it "pipeline" is kind of misleading, because the actual "piping" (connecting processes to each other) is handled elsewhere. In fact, using your "pipeline" abstraction, you could perfectly well run a bunch of processes that aren't connected to each other at all. Is my understanding correct?
- seliopou 11y agoFor those that are interested in libraries like these, OCaml has an implementation that written by Jane Street called Pipes[0]. It's been around for years, helps trade billions of dollars a day, is written in a language that actually has parametric polymorphism, and a configurable notion of "pushback". I've written about it before[1]. [0]: https://ocaml.janestreet.com/ocaml-core/111.28.00/doc/async_kernel/#Pipe https://ocaml.janestreet.com/ocaml-core/111.28.00/doc/async_... [1]: https://news.ycombinator.com/threads?id=seliopou&next=9809300 https://news.ycombinator.com/threads?id=seliopou&next=980930...
- samuell 11y agoThanks much for the links! What do you mean with "pushback"? Is it the same as "back-pressure"? (If so, Go has "implicit backpressure" in its bounded buffers on the channels ... although I don't think it is changeable at runtime, unless you replace the channel ... but not 100% sure ...)
- PopsiclePete 11y agoA lot of cool concepts come from languages like OCaml and Haskell. But for better or worse, we live in a world where simple languages rule the day (C++ being one notable exception), and if we get lucky, we get some of cool OCaml/Forth/Haskell/LISP/Erlang features to play with, in our simple little worlds with our simple little toy languages like Java and C# and Go and Javascript... it's sad in a way, but it's reality. I'd much rather write F# (basically .NET-hosted OCaml) than C#, but no luck finding gainful employment with that...
- samuell 11y agoBtw, I should also give a shoutout to Vladimir Vivien's Automi library, also posted on the GopherAcademy blog some days ago: https://blog.gopheracademy.com/advent-2015/automi-stream-processing-over-go-channels https://blog.gopheracademy.com/advent-2015/automi-stream-pro... It takes an interesting, slightly different approach, in my view especially for slightly lower-level operations compared to the "scipipe" pattern above. For example, it looks like something I'd use for implementing the internals of components in my pattern.
- whateveracct 11y agoscalaz-stream does all this and way more and way better while actually being "composable"