12 ms·

Flow-Based Programming

217 points by hypomnemata 6y ago

devmunchies 6y ago
Similar concept to what I do when programming in F#/OCaml by default. You create your data types and then they "flow" through functions. Each function is `input -> output` but bigger workflow are also input -> output.
- adamnemecek 6y ago
  The main difference is that with data flow, the flows happen conceptually at the same time.
hpoe 6y ago
As I was reading this I was enthusiastically agreeing with the idea, but something felt super familiar about it. Then I realized this is basically the same concept as Unix pipes. I have my little individual programs, grep, awk, sed, jq, etc, and then I can endlessly mix and match those different components to do what I want. The limitation that I have seen with Unix pipes isn't often with the ability to process or manage the data it is that it only works if all the data is setup the proper way. As I have been in the industry longer it seems to me that most code that gets written isn't about actual computing but just centered around importing and transforming data that is expressed in different ways. Is there something that makes reading in disparate data records easier so I can focus on the computing part of computer program and less time on parsing data?
- adamnemecek 6y ago
  Unix pipes make it hard to have a large graph. Like most pipes take one thing and produce one thing. You don’t really have complex data flow graphs.
  - taeric 6y ago
    Though, this really kind of fits with the metaphor. Pipes typically just send liquid. You could have a conduit connector to join a bunch of wires/signals, I suppose. But, realistically, that is a mess always.
    - adamnemecek 6y ago
      Right but you generally have a complex pipe system even with liquids. Sewers send waste to a particular processing facility. A node editor makes it somewhat less messy.
  - lispweeniejr 6y ago
    you aren't wrong, but i wanted to point out that through `tee` and `cat` you can split or merge pipelines, and with FIFOs you can awkwardly create graph cycles
phreeza 6y ago
Apache Beam seems to be an implementation of this idea. It works well when the things you have to do matches the logic, but it gets tricky if you need to do stuff iteratively or recursively.
- samuell 6y ago
  Beam has some overlap, but in my understanding it has a rather involved syntax for defining the data flows, quite far from the simple list of connections between in and out ports that you see in FBP systems following J P Morrisons principles more closely. Apache NiFi comes quite a lot closer, with the main difference that they only have a single in-port, instead of separate named ones. Also it seems to be among the more heavy and complex implementations (for good and bad).
  - jpaulmorrison 6y ago
    "They" meaning "FBP-like" or "FBP-inspired systems", to use Joe Witt's terminology... They are usually control-flow oriented, and to my mind do not yield the productivity and maintainability advantages of the "classical" FBP paradigm shift. I recently asked Node-RED's Nick O'Leary why Node-RED only allows one input port, and the answer was something like "because we have never run into the requirement"... a) this is not something that can easily be added after the fact, without totally rethinking the product architecture, and b) trying to build complex systems without that feature would be, to me, like trying to hang wallpaper with only one hand! One litmus test I use is that of concatenating data streams using the two different paradigms - I tried to describe this in an article a year or so ago, recently updated: https://jpaulm.github.io/fbp/concat.html https://jpaulm.github.io/fbp/concat.html Cheers!
bergie 6y ago
Nice to see this here, as it has been the inspiration for quite a few years of my work: https://noflojs.org/ https://noflojs.org/
- leetrout 6y ago
  NoFlo really is impressive. Great work.
- bmitc 6y ago
  Do you have a shining example of someone using NoFlo? It doesn't have to be big, but one that you think really exemplifies what it excels at. (By the way, I'm a big fan of visual programming, so I'm just curious.)
  - bergie 6y ago
    There has been quite a lot of IoT automation / rule engine stuff done in NoFlo in the last couple of companies I’ve worked at, but obviously I can’t really go into specifics there. Then there was this startup way back when: https://www.cloudamqp.com/blog/2017-05-31-work-queues-in-rabbitmq-for-resource-intensive-tasks.html https://www.cloudamqp.com/blog/2017-05-31-work-queues-in-rab... https://librearts.org/2014/10/artificial-intelligence-designs-websites-uses-open-technology-stack/ https://librearts.org/2014/10/artificial-intelligence-design... On the front-end side, Flowhub uses NoFlo for state management, a bit similarly to how you’d use Redux: https://github.com/noflo/noflo-ui https://github.com/noflo/noflo-ui
    - bmitc 6y ago
      Thanks for the links! I’ll check them out.
amelius 6y ago
Flow-based programming is cool, until the flow starts altering the flow.
- cjohnson318 6y ago
  I think at that point it's a complex dynamic system.
samuell 6y ago
The core of the FBP principles are the holy grail of true componentized function architecture. Instead of losing yourself in ever more complex syntax convolutions as has happened in a lot of functional programming, you make the components (even long running stateful ones) self contained with ports for data input and output as the sole means of communicating with them, over buffered channels to allow asynchronous computation, and most importantly keep the network definition separate. Just this idea in itself is just brilliant. Hats off to Mr Morrison for that! It allows to decouple complex software into reusable components, without clever FP syntax. (Though, FP is perfect for implementing the components themselves. It just doesn't really scale all too well for whole program architecture, in my experience). One point to note: The visual component of many fbp systems is completely optional, as is the idea of using novel DSLs. You can as well define your networks and components in pure code. See GoFlow https://github.com/trustmaster/goflow https://github.com/trustmaster/goflow) and my own little experiment FlowBase (https://flowbase.org https://flowbase.org) for examples of that, in Go. I successfully built a rather complex little app to convert from Semantic web RDF format to (semantic) mediawiki XML dump format, in two weeks straight, of linear development time: for each component (of ca 7), implement, test, go to the next component (See: https://github.com/rdfio/rdf2smw https://github.com/rdfio/rdf2smw) The same implementation in procedural PHP took months, and still doesn't have all bugs and strange behaviours filed out.
- the_duke 6y ago
  > you make the components self contained with ports for data input and output as the sole means of communicating with them, over buffered channels to allow asynchronous computation This concept sounds exactly like actor systems like Erlang/OTP and Akka, only with a different set of terminology. The submitted site and your comment don't mention those anywhere. Are there appreciable differences between actor systems and FBP?
  - megameter 6y ago
    FBP is most analogous to the unit record machines of yore: https://en.m.wikipedia.org/wiki/Unit_record_equipment https://en.m.wikipedia.org/wiki/Unit_record_equipment What you are programming is the graph of how these machines connect and flow data(keypunch cards) through each other, while the machines themselves and their function are abstracted out. Data does not stay "at rest", it's presumed to reach a terminating point where it exits the graph. A buildup of unprocessed data in a machine's inbox results in an overflow. It's a very useful model for making a debuggable asynchronous system since it imposes static constraints everywhere that you can map to your real hardware resources, versus the emphasis on dynamism seen in the actor model(actors hold private data, modify their state, create new actors - all explicitly disallowed in FBP).
akavel 6y ago
Mandatory fanboi mention: https://luna-lang.org/ https://luna-lang.org/ - now Enso: https://medium.com/@enso_org/enso-dev-blog-18th-december-2020-e51e11c02c66 https://medium.com/@enso_org/enso-dev-blog-18th-december-202...
- samuell 6y ago
  I'm super intrigued by luna/enso. Only that every time I've tried it, the editor has had serious stability problems. I wonder if a more traditional tooling and editor support wouldn't be more successful for wide adoption?
- pantsforbirds 6y ago
  Luna is one of the coolest projects I've seen. I've really enjoyed watching the progress. Curious to see if it (or concepts from it) gets picked up for programming education one day.
fahrrad34 6y ago
Do I understand it correct when I think that clojure’s core.async is an example implementation?
- dustingetz 6y ago
  not quite, but this seems close: https://github.com/leonoel/missionary https://github.com/leonoel/missionary
- adamkl 6y ago
  Rich Hickey gave a good talk on core.async and it does seem pretty similar in concept: https://youtu.be/drmNlZVkUeE https://youtu.be/drmNlZVkUeE
teknopurge 6y ago
https://nodered.org/ https://nodered.org/ - great project for all sorts of needs.
- jcims 6y ago
  I recently used nodered for some process automation and it was absolutely fantastic for quickly prototyping, dashboarding, doing sensor integration, etc. Highly recommended!!!
jgraettinger1 6y ago
We're building a tool, Estuary Flow, which seeks to be an end-to-end realization of practical, configuration driven, and scale-out flow-based programming -- with an important twist. The central concept is a "collection", an append-only set of schematized documents, which can be captured and materialized into other systems (e.x. pub/sub, S3 buckets, etc). "Derivations" are collections defined in terms of source collections, and stateful transformations/joins/aggregations which are applied to them. A key twist is that collections are simultaneously a batch dataset (backed by cloud-storage) and also a real-time stream. They unify the current dichotomy of "historical" vs "streaming" data into a single addressed concept. Declare a new derivation, and it automatically back-fills over history right from S3, then seamlessly transitions to live data. If this sounds interesting, check out our docs [0]. We're early, but love feedback! [0] https://estuary.readthedocs.io/en/latest/README.html https://estuary.readthedocs.io/en/latest/README.html
ledauphin 6y ago
I don't mean to derail the conversation, but this really does remind me of the game Factorio, though sort of in reverse. In Factorio, you build a larger and larger factory out of pre-established functional components (assemblers, labs, chemical plants, etc) that take in a limited set of inputs and produce (usually) a single output. Your challenge is not to define the functional core processes, but instead to wire together those functional components by connecting their inputs and outputs in ever-more-automated fashion, starting by hand, then using simple belts (pipes) that eventually allow arbitrary load-balancing via "splitters", and eventually through to trains (the forking and load balancing happening via backpressure in the train system) and robots (where everything is managed essentially as a single state database of requests, and backpressure is provided by output limitations, usually per functional component). Naively, I think that someday a decent chunk of programming might actually look like this, and parts even be represented visually (though in my opinion likely still defined formally as text). Only I think programmers will continue to write the functional components themselves, unlike in Factorio. They'll just live on different levels of the "codebase", and the "pipes" level will likely be a lot more abstracted than it is in Factorio. As a software developer, I find this paradigm to map very well to serverless architectures, because you generally want to think a level higher than the per-machine basis. It does require a willingness to forgo handy and well-established tools like the filesystem and Unix pipes in favor of higher level abstractions around transfer and storage of data.
- freeqaz 6y ago
  You have, from first principles, reconstructed a huge portion of my thought process for building https://refinery.io https://refinery.io Factorio and Minecraft automation mods are a big inspiration! Check out InfiniFactory too :) Bridging existing applications to the Serverless paradigm is far from simple. That's one of the biggest struggles I've experienced trying to build a Flow-based software platform. Learn more every day though. Thank you for the interesting comment!
  - mikewarot 6y ago
    I took a look at your site (refinery.io), and the "watch demo" is really a "read introduction"... I was expecting to sit back and let you show me 10-15 minutes of video that sells the idea. It looks very professionally done, but reading screens isn't as easy as it used to be for me, a video is better. Good luck!
voldacar 6y ago
This is a good article, in the history or multithreading sections it would have been nice to mention the various old hardware implementations of this model, such as the transputer or connection machine
analog31 6y ago
It might not tick all of the boxes for being a complete software development tool, but Excel strikes me as being a dataflow programming model. I wonder if it's a reason why it's easy for laypeople to learn.
Geminidog 6y ago
This type of programming is actually a subset of functional programming called point free programming. It's equivalent to programming using only combinators as the fundamental unit. https://en.wikipedia.org/wiki/Tacit_programming https://en.wikipedia.org/wiki/Tacit_programming All programs are actually pipelines of data flowing from a low entropy state to high entropy state with IO and state as the endpoints of the pipes-. Using the point free style or flow based programming makes this entropy and the pipelined nature of all programs more explicit. Using OOP or regular programming the pipeline nature of data flowing to and from state and IO becomes less evident and more convoluted.
mikewarot 6y ago
Side effects are forbidden by structure, flows could be monitored in a GUI/Debugger, and as a result components can be tested as a unit, instead of a whole system. I love it! It is easier to design digital circuits when you have a whole catalog of 7400 and 4000 series gates, than it is using individual transistors. It is easier to wire a house when you're not making wires and switches with a hammer and a forge. I welcome this new higher level of abstraction, and am willing to pay the cost in terms of CPU and Memory to get there, just as I'm willing to waste transistors or copper wire to have something done and working.
dang 6y ago
If curious see also from 2015, on the same article: https://news.ycombinator.com/item?id=8867584 https://news.ycombinator.com/item?id=8867584 Related: 2019 (a bit) https://news.ycombinator.com/item?id=20215592 https://news.ycombinator.com/item?id=20215592 2019 (another bit) https://news.ycombinator.com/item?id=19203642 https://news.ycombinator.com/item?id=19203642 2019 https://news.ycombinator.com/item?id=18859019 https://news.ycombinator.com/item?id=18859019 2015 (not that good) https://news.ycombinator.com/item?id=10755250 https://news.ycombinator.com/item?id=10755250 2015 (a bit better) https://news.ycombinator.com/item?id=9718868 https://news.ycombinator.com/item?id=9718868 2015 https://news.ycombinator.com/item?id=8992281 https://news.ycombinator.com/item?id=8992281 2013 https://news.ycombinator.com/item?id=6264657 https://news.ycombinator.com/item?id=6264657 2012 https://news.ycombinator.com/item?id=4239274 https://news.ycombinator.com/item?id=4239274
- homieg33 6y ago
  Very useful list. Thanks for providing.
pantsforbirds 6y ago
You can write pretty interesting data pipelines using akka streams with a very similar idea to FBP. It's probably not technically FBP, but the whole reactive stream implementation is similar in thought but allows for things like disparate data source speeds without blowing out part of the flow.
hexo 6y ago
This reminds me of reactive programming, especially functional reactive programming.
u678u 6y ago
Its great but surely its more like OO programming perhaps without inheritance.
galaxyLogic 6y ago
In Dataflow -programming (or something like that) how do you program a decision that depends on the result of some component? It would seem like I need to suspend my computation then "ask" the result from some component, get the result back, and then alter my computation based on that result. Pure flow-forward would not seem to support this easily. Or can it? Or does it come down to that we always will need BOTH sync- and async- functions? (unless of course we limit the problem domain)
- erichocean 6y ago
  Spreadsheets are a variant of Dataflow programming, so however they do it.
  - galaxyLogic 6y ago
    Right but spreadsheets are hardly used for "general purpose programming". They are good for the tasks they are used for but "systems" are not implemented with spreadsheets.
    - erichocean 6y ago
      If you are implying that normal, for+while loop programming is difficult with dataflow approaches…you're right.
      - galaxyLogic 6y ago
        That's what I would suspect. Dataflow programing would seem to hold great promise in making programs more understandable. But there would seem to be limits as to where it can be applied (for-if) etc. It is not a universal model for practical programming, I assume. It's a bit like "pure functional programming". We can't really do that in practice because we need IO. The best we can do is to divide the program into two parts, purely functional, and imperative. Similarly I assume we could (and should try to) divide programs into "data-flow-part" and "synchronous-part".
- dustingetz 6y ago
smartmic 6y ago
A nice and powerful C++ flow-based programming framework is DSPatch [1]. It is definitely worth to mention here, check out also the examples in the audio domain. [1] http://flowbasedprogramming.com/docs/html/index.html http://flowbasedprogramming.com/docs/html/index.html
blackrock 6y ago
Fascinating. This is how I build my programs. I never knew it had a particular name for the style. I just assumed it was functional-style programming. This allows me to build ever greater programs, that does very complicated things, and has thousands of lines of code and logic, but everything gets boiled down to small bite-sized pieces. It’s a more data oriented approach. And it seems to follow an assembly line model. I liken it to building code pyramids. Where I keep stacking and chaining one function to another, to build even bigger pyramids. At the end, all the code is heavily unit tested, and I have a high degree of trust in the fidelity of the codebase.
- 0x445442 6y ago
  Yeah, when Java 8 introduced the Function interface I started to program this way as well. I get push back some times from the those not initiated to the functional style because I have a ton of little function that do one thing. Unfortunately in Java this means a bunch of class files. The other thing I noticed when I started using this style is that I was able to ditch unit testing mocking frameworks. This made me realize, when the Function interface was introduce it was a backdoor way to get folks back to coding to an interface, which we should have been doing all along.
jes5199 6y ago
neat to see this here! I've recently become convinced Flow Based Programming is the ideal that other programming paradigms are reaching for. But in practice, there's a lot that's not obvious to me how it would actually work-- so it's great to have a bunch of new reading suggestions!
golemotron 6y ago
Thank you, Doug McIlroy.
tyingq 6y ago
The analogy to the soft drink bottling seems...off. I would imagine there's a contract of sorts between stations on rate of output/input, spacing/placement of bottles, etc, between stations that makes it not very asynchronous.
deleted 6y ago
[deleted]
cs404 6y ago
What is the difference from flow and event driven programming
Ericson2314 6y ago
This is wildly under developed, mathematically, but eventually with enough FP and category theory and things, we will get back there. The problem is everyone wants to write little nodes and "just" wire them up. But all the real complexity is not in the nodes, but the nature of the wires and their composition. We'll get there though, stay tuned.
ngneer 6y ago
Sounds like TPL Dataflow
pgt 6y ago
Honourable mention for Petri Nets: https://en.wikipedia.org/wiki/Petri_net https://en.wikipedia.org/wiki/Petri_net
mujina93 6y ago
Have a look at the Oz PL and its canonical textbook: https://en.m.wikipedia.org/wiki/Oz_(programming_language) https://en.m.wikipedia.org/wiki/Oz_(programming_language)
wyqydsyq 6y ago
This paradigm reminds me a lot of my favourite JS/TS framework at the moment: Cycle.JS https://cycle.js.org/ https://cycle.js.org/
lincpa 6y ago
The Grand Unified Programming Theory: The Pure Function Pipeline Data Flow with principle-based Warehouse/Workshop Model https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow
jpaulmorrison 6y ago
General comment about the Wikipedia article - https://en.wikipedia.org/wiki/Flow-based_programming https://en.wikipedia.org/wiki/Flow-based_programming : given all the interesting discussion on the 'Net, including this thread, it might be time to update the Wikipedia article. The most substantive addition in recent years is Akaigoro's comment about Actors, added Jan. 2020 (@guitarvydas, care to jump in?!), preceeded, I think, by Joe Witt's reference to NiFi in mid-2018. This general lack of activity I feel might lead readers to assume that FBP is an outdated concept, whereas this thread proves it's definitely alive and kicking! I, personally, am not allowed to update the article, due to the WP ban on self-promotion, so I would like to encourage people to add topics, controversies, anything, to the WP article... Freshen it up a bit, as it were! Thanks, and stay safe, everyone!

217 points by hypomnemata 6y ago

Similar concept to what I do when programming in F#/OCaml by default. You create your data types and then they "flow" through functions. Each function is `input -> output` but bigger workflow are also input -> output.

adamnemecek 6y ago
The main difference is that with data flow, the flows happen conceptually at the same time.

hpoe 6y ago

As I was reading this I was enthusiastically agreeing with the idea, but something felt super familiar about it. Then I realized this is basically the same concept as Unix pipes. I have my little individual programs, grep, awk, sed, jq, etc, and then I can endlessly mix and match those different components to do what I want. The limitation that I have seen with Unix pipes isn't often with the ability to process or manage the data it is that it only works if all the data is setup the proper way. As I have been in the industry longer it seems to me that most code that gets written isn't about actual computing but just centered around importing and transforming data that is expressed in different ways. Is there something that makes reading in disparate data records easier so I can focus on the computing part of computer program and less time on parsing data?

adamnemecek 6y ago
Unix pipes make it hard to have a large graph. Like most pipes take one thing and produce one thing. You don’t really have complex data flow graphs.
- taeric 6y ago
  Though, this really kind of fits with the metaphor. Pipes typically just send liquid. You could have a conduit connector to join a bunch of wires/signals, I suppose. But, realistically, that is a mess always.
  - adamnemecek 6y ago
    Right but you generally have a complex pipe system even with liquids. Sewers send waste to a particular processing facility. A node editor makes it somewhat less messy.
- lispweeniejr 6y ago
  you aren't wrong, but i wanted to point out that through `tee` and `cat` you can split or merge pipelines, and with FIFOs you can awkwardly create graph cycles

phreeza 6y ago

Apache Beam seems to be an implementation of this idea. It works well when the things you have to do matches the logic, but it gets tricky if you need to do stuff iteratively or recursively.

samuell 6y ago
Beam has some overlap, but in my understanding it has a rather involved syntax for defining the data flows, quite far from the simple list of connections between in and out ports that you see in FBP systems following J P Morrisons principles more closely. Apache NiFi comes quite a lot closer, with the main difference that they only have a single in-port, instead of separate named ones. Also it seems to be among the more heavy and complex implementations (for good and bad).
- jpaulmorrison 6y ago
  "They" meaning "FBP-like" or "FBP-inspired systems", to use Joe Witt's terminology... They are usually control-flow oriented, and to my mind do not yield the productivity and maintainability advantages of the "classical" FBP paradigm shift. I recently asked Node-RED's Nick O'Leary why Node-RED only allows one input port, and the answer was something like "because we have never run into the requirement"... a) this is not something that can easily be added after the fact, without totally rethinking the product architecture, and b) trying to build complex systems without that feature would be, to me, like trying to hang wallpaper with only one hand! One litmus test I use is that of concatenating data streams using the two different paradigms - I tried to describe this in an article a year or so ago, recently updated: https://jpaulm.github.io/fbp/concat.html https://jpaulm.github.io/fbp/concat.html Cheers!

bergie 6y ago

Nice to see this here, as it has been the inspiration for quite a few years of my work: https://noflojs.org/ https://noflojs.org/

leetrout 6y ago
NoFlo really is impressive. Great work.
bmitc 6y ago
Do you have a shining example of someone using NoFlo? It doesn't have to be big, but one that you think really exemplifies what it excels at. (By the way, I'm a big fan of visual programming, so I'm just curious.)
- bergie 6y ago
  There has been quite a lot of IoT automation / rule engine stuff done in NoFlo in the last couple of companies I’ve worked at, but obviously I can’t really go into specifics there. Then there was this startup way back when: https://www.cloudamqp.com/blog/2017-05-31-work-queues-in-rabbitmq-for-resource-intensive-tasks.html https://www.cloudamqp.com/blog/2017-05-31-work-queues-in-rab... https://librearts.org/2014/10/artificial-intelligence-designs-websites-uses-open-technology-stack/ https://librearts.org/2014/10/artificial-intelligence-design... On the front-end side, Flowhub uses NoFlo for state management, a bit similarly to how you’d use Redux: https://github.com/noflo/noflo-ui https://github.com/noflo/noflo-ui
  - bmitc 6y ago
    Thanks for the links! I’ll check them out.

amelius 6y ago

Flow-based programming is cool, until the flow starts altering the flow.

cjohnson318 6y ago
I think at that point it's a complex dynamic system.

samuell 6y ago

The core of the FBP principles are the holy grail of true componentized function architecture. Instead of losing yourself in ever more complex syntax convolutions as has happened in a lot of functional programming, you make the components (even long running stateful ones) self contained with ports for data input and output as the sole means of communicating with them, over buffered channels to allow asynchronous computation, and most importantly keep the network definition separate. Just this idea in itself is just brilliant. Hats off to Mr Morrison for that! It allows to decouple complex software into reusable components, without clever FP syntax. (Though, FP is perfect for implementing the components themselves. It just doesn't really scale all too well for whole program architecture, in my experience). One point to note: The visual component of many fbp systems is completely optional, as is the idea of using novel DSLs. You can as well define your networks and components in pure code. See GoFlow https://github.com/trustmaster/goflow https://github.com/trustmaster/goflow) and my own little experiment FlowBase (https://flowbase.org https://flowbase.org) for examples of that, in Go. I successfully built a rather complex little app to convert from Semantic web RDF format to (semantic) mediawiki XML dump format, in two weeks straight, of linear development time: for each component (of ca 7), implement, test, go to the next component (See: https://github.com/rdfio/rdf2smw https://github.com/rdfio/rdf2smw) The same implementation in procedural PHP took months, and still doesn't have all bugs and strange behaviours filed out.

the_duke 6y ago
> you make the components self contained with ports for data input and output as the sole means of communicating with them, over buffered channels to allow asynchronous computation This concept sounds exactly like actor systems like Erlang/OTP and Akka, only with a different set of terminology. The submitted site and your comment don't mention those anywhere. Are there appreciable differences between actor systems and FBP?
- megameter 6y ago
  FBP is most analogous to the unit record machines of yore: https://en.m.wikipedia.org/wiki/Unit_record_equipment https://en.m.wikipedia.org/wiki/Unit_record_equipment What you are programming is the graph of how these machines connect and flow data(keypunch cards) through each other, while the machines themselves and their function are abstracted out. Data does not stay "at rest", it's presumed to reach a terminating point where it exits the graph. A buildup of unprocessed data in a machine's inbox results in an overflow. It's a very useful model for making a debuggable asynchronous system since it imposes static constraints everywhere that you can map to your real hardware resources, versus the emphasis on dynamism seen in the actor model(actors hold private data, modify their state, create new actors - all explicitly disallowed in FBP).

akavel 6y ago

Mandatory fanboi mention: https://luna-lang.org/ https://luna-lang.org/ - now Enso: https://medium.com/@enso_org/enso-dev-blog-18th-december-2020-e51e11c02c66 https://medium.com/@enso_org/enso-dev-blog-18th-december-202...

samuell 6y ago
I'm super intrigued by luna/enso. Only that every time I've tried it, the editor has had serious stability problems. I wonder if a more traditional tooling and editor support wouldn't be more successful for wide adoption?
pantsforbirds 6y ago
Luna is one of the coolest projects I've seen. I've really enjoyed watching the progress. Curious to see if it (or concepts from it) gets picked up for programming education one day.

fahrrad34 6y ago

Do I understand it correct when I think that clojure’s core.async is an example implementation?

dustingetz 6y ago
not quite, but this seems close: https://github.com/leonoel/missionary https://github.com/leonoel/missionary
adamkl 6y ago
Rich Hickey gave a good talk on core.async and it does seem pretty similar in concept: https://youtu.be/drmNlZVkUeE https://youtu.be/drmNlZVkUeE

teknopurge 6y ago

https://nodered.org/ https://nodered.org/ - great project for all sorts of needs.

jcims 6y ago
I recently used nodered for some process automation and it was absolutely fantastic for quickly prototyping, dashboarding, doing sensor integration, etc. Highly recommended!!!

jgraettinger1 6y ago

We're building a tool, Estuary Flow, which seeks to be an end-to-end realization of practical, configuration driven, and scale-out flow-based programming -- with an important twist. The central concept is a "collection", an append-only set of schematized documents, which can be captured and materialized into other systems (e.x. pub/sub, S3 buckets, etc). "Derivations" are collections defined in terms of source collections, and stateful transformations/joins/aggregations which are applied to them. A key twist is that collections are simultaneously a batch dataset (backed by cloud-storage) and also a real-time stream. They unify the current dichotomy of "historical" vs "streaming" data into a single addressed concept. Declare a new derivation, and it automatically back-fills over history right from S3, then seamlessly transitions to live data. If this sounds interesting, check out our docs [0]. We're early, but love feedback! [0] https://estuary.readthedocs.io/en/latest/README.html https://estuary.readthedocs.io/en/latest/README.html

ledauphin 6y ago

I don't mean to derail the conversation, but this really does remind me of the game Factorio, though sort of in reverse. In Factorio, you build a larger and larger factory out of pre-established functional components (assemblers, labs, chemical plants, etc) that take in a limited set of inputs and produce (usually) a single output. Your challenge is not to define the functional core processes, but instead to wire together those functional components by connecting their inputs and outputs in ever-more-automated fashion, starting by hand, then using simple belts (pipes) that eventually allow arbitrary load-balancing via "splitters", and eventually through to trains (the forking and load balancing happening via backpressure in the train system) and robots (where everything is managed essentially as a single state database of requests, and backpressure is provided by output limitations, usually per functional component). Naively, I think that someday a decent chunk of programming might actually look like this, and parts even be represented visually (though in my opinion likely still defined formally as text). Only I think programmers will continue to write the functional components themselves, unlike in Factorio. They'll just live on different levels of the "codebase", and the "pipes" level will likely be a lot more abstracted than it is in Factorio. As a software developer, I find this paradigm to map very well to serverless architectures, because you generally want to think a level higher than the per-machine basis. It does require a willingness to forgo handy and well-established tools like the filesystem and Unix pipes in favor of higher level abstractions around transfer and storage of data.

freeqaz 6y ago
You have, from first principles, reconstructed a huge portion of my thought process for building https://refinery.io https://refinery.io Factorio and Minecraft automation mods are a big inspiration! Check out InfiniFactory too :) Bridging existing applications to the Serverless paradigm is far from simple. That's one of the biggest struggles I've experienced trying to build a Flow-based software platform. Learn more every day though. Thank you for the interesting comment!
- mikewarot 6y ago
  I took a look at your site (refinery.io), and the "watch demo" is really a "read introduction"... I was expecting to sit back and let you show me 10-15 minutes of video that sells the idea. It looks very professionally done, but reading screens isn't as easy as it used to be for me, a video is better. Good luck!

voldacar 6y ago

This is a good article, in the history or multithreading sections it would have been nice to mention the various old hardware implementations of this model, such as the transputer or connection machine

analog31 6y ago

It might not tick all of the boxes for being a complete software development tool, but Excel strikes me as being a dataflow programming model. I wonder if it's a reason why it's easy for laypeople to learn.

Geminidog 6y ago

This type of programming is actually a subset of functional programming called point free programming. It's equivalent to programming using only combinators as the fundamental unit. https://en.wikipedia.org/wiki/Tacit_programming https://en.wikipedia.org/wiki/Tacit_programming All programs are actually pipelines of data flowing from a low entropy state to high entropy state with IO and state as the endpoints of the pipes-. Using the point free style or flow based programming makes this entropy and the pipelined nature of all programs more explicit. Using OOP or regular programming the pipeline nature of data flowing to and from state and IO becomes less evident and more convoluted.

mikewarot 6y ago

Side effects are forbidden by structure, flows could be monitored in a GUI/Debugger, and as a result components can be tested as a unit, instead of a whole system. I love it! It is easier to design digital circuits when you have a whole catalog of 7400 and 4000 series gates, than it is using individual transistors. It is easier to wire a house when you're not making wires and switches with a hammer and a forge. I welcome this new higher level of abstraction, and am willing to pay the cost in terms of CPU and Memory to get there, just as I'm willing to waste transistors or copper wire to have something done and working.

dang 6y ago

If curious see also from 2015, on the same article: https://news.ycombinator.com/item?id=8867584 https://news.ycombinator.com/item?id=8867584 Related: 2019 (a bit) https://news.ycombinator.com/item?id=20215592 https://news.ycombinator.com/item?id=20215592 2019 (another bit) https://news.ycombinator.com/item?id=19203642 https://news.ycombinator.com/item?id=19203642 2019 https://news.ycombinator.com/item?id=18859019 https://news.ycombinator.com/item?id=18859019 2015 (not that good) https://news.ycombinator.com/item?id=10755250 https://news.ycombinator.com/item?id=10755250 2015 (a bit better) https://news.ycombinator.com/item?id=9718868 https://news.ycombinator.com/item?id=9718868 2015 https://news.ycombinator.com/item?id=8992281 https://news.ycombinator.com/item?id=8992281 2013 https://news.ycombinator.com/item?id=6264657 https://news.ycombinator.com/item?id=6264657 2012 https://news.ycombinator.com/item?id=4239274 https://news.ycombinator.com/item?id=4239274

homieg33 6y ago
Very useful list. Thanks for providing.

pantsforbirds 6y ago

You can write pretty interesting data pipelines using akka streams with a very similar idea to FBP. It's probably not technically FBP, but the whole reactive stream implementation is similar in thought but allows for things like disparate data source speeds without blowing out part of the flow.

hexo 6y ago

This reminds me of reactive programming, especially functional reactive programming.

u678u 6y ago

Its great but surely its more like OO programming perhaps without inheritance.

galaxyLogic 6y ago

In Dataflow -programming (or something like that) how do you program a decision that depends on the result of some component? It would seem like I need to suspend my computation then "ask" the result from some component, get the result back, and then alter my computation based on that result. Pure flow-forward would not seem to support this easily. Or can it? Or does it come down to that we always will need BOTH sync- and async- functions? (unless of course we limit the problem domain)

erichocean 6y ago
Spreadsheets are a variant of Dataflow programming, so however they do it.
- galaxyLogic 6y ago
  Right but spreadsheets are hardly used for "general purpose programming". They are good for the tasks they are used for but "systems" are not implemented with spreadsheets.
  - erichocean 6y ago
    If you are implying that normal, for+while loop programming is difficult with dataflow approaches…you're right.
    - galaxyLogic 6y ago
      That's what I would suspect. Dataflow programing would seem to hold great promise in making programs more understandable. But there would seem to be limits as to where it can be applied (for-if) etc. It is not a universal model for practical programming, I assume. It's a bit like "pure functional programming". We can't really do that in practice because we need IO. The best we can do is to divide the program into two parts, purely functional, and imperative. Similarly I assume we could (and should try to) divide programs into "data-flow-part" and "synchronous-part".
dustingetz 6y ago

smartmic 6y ago

A nice and powerful C++ flow-based programming framework is DSPatch [1]. It is definitely worth to mention here, check out also the examples in the audio domain. [1] http://flowbasedprogramming.com/docs/html/index.html http://flowbasedprogramming.com/docs/html/index.html

blackrock 6y ago

Fascinating. This is how I build my programs. I never knew it had a particular name for the style. I just assumed it was functional-style programming. This allows me to build ever greater programs, that does very complicated things, and has thousands of lines of code and logic, but everything gets boiled down to small bite-sized pieces. It’s a more data oriented approach. And it seems to follow an assembly line model. I liken it to building code pyramids. Where I keep stacking and chaining one function to another, to build even bigger pyramids. At the end, all the code is heavily unit tested, and I have a high degree of trust in the fidelity of the codebase.

0x445442 6y ago
Yeah, when Java 8 introduced the Function interface I started to program this way as well. I get push back some times from the those not initiated to the functional style because I have a ton of little function that do one thing. Unfortunately in Java this means a bunch of class files. The other thing I noticed when I started using this style is that I was able to ditch unit testing mocking frameworks. This made me realize, when the Function interface was introduce it was a backdoor way to get folks back to coding to an interface, which we should have been doing all along.

jes5199 6y ago

neat to see this here! I've recently become convinced Flow Based Programming is the ideal that other programming paradigms are reaching for. But in practice, there's a lot that's not obvious to me how it would actually work-- so it's great to have a bunch of new reading suggestions!

golemotron 6y ago

Thank you, Doug McIlroy.

tyingq 6y ago

The analogy to the soft drink bottling seems...off. I would imagine there's a contract of sorts between stations on rate of output/input, spacing/placement of bottles, etc, between stations that makes it not very asynchronous.

deleted 6y ago

[deleted]

cs404 6y ago

What is the difference from flow and event driven programming

Ericson2314 6y ago

This is wildly under developed, mathematically, but eventually with enough FP and category theory and things, we will get back there. The problem is everyone wants to write little nodes and "just" wire them up. But all the real complexity is not in the nodes, but the nature of the wires and their composition. We'll get there though, stay tuned.

ngneer 6y ago

Sounds like TPL Dataflow

pgt 6y ago

Honourable mention for Petri Nets: https://en.wikipedia.org/wiki/Petri_net https://en.wikipedia.org/wiki/Petri_net

mujina93 6y ago

Have a look at the Oz PL and its canonical textbook: https://en.m.wikipedia.org/wiki/Oz_(programming_language) https://en.m.wikipedia.org/wiki/Oz_(programming_language)

wyqydsyq 6y ago

This paradigm reminds me a lot of my favourite JS/TS framework at the moment: Cycle.JS https://cycle.js.org/ https://cycle.js.org/

lincpa 6y ago

The Grand Unified Programming Theory: The Pure Function Pipeline Data Flow with principle-based Warehouse/Workshop Model https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow

jpaulmorrison 6y ago

General comment about the Wikipedia article - https://en.wikipedia.org/wiki/Flow-based_programming https://en.wikipedia.org/wiki/Flow-based_programming : given all the interesting discussion on the 'Net, including this thread, it might be time to update the Wikipedia article. The most substantive addition in recent years is Akaigoro's comment about Actors, added Jan. 2020 (@guitarvydas, care to jump in?!), preceeded, I think, by Joe Witt's reference to NiFi in mid-2018. This general lack of activity I feel might lead readers to assume that FBP is an outdated concept, whereas this thread proves it's definitely alive and kicking! I, personally, am not allowed to update the article, due to the WP ban on self-promotion, so I would like to encourage people to add topics, controversies, anything, to the WP article... Freshen it up a bit, as it were! Thanks, and stay safe, everyone!