7 ms·
An introduction to NoFlo and flow-based programming
- SomeCallMeTim 12y agoI think the types of problems each programmer tends to confront make up a large part of the disconnect between the ones who "drink the koolaid" of a programming paradigm or process and the people who are unimpressed. I typically do games. Games are useful to think, in part, in OOP terms; there is very little "data flow" and (typically) there are a lot of objects that need to interact in asynchronous ways. There are specific parts that could be reduced to flow based programming (FBP), but it seems like the wrong paradigm for writing an entire game, as an example close to me. I could imagine FBP certain AI logic state machines being useful, for instance. But other areas of development have flow-style tools and that's clearly the right way to write code. Signal processing is an obvious example: When you want to send an audio signal through a reverb, split it to two different types of processors and re-merge it later with a fader tied to an external control, laying those components out with boxes is 1000x better than trying to write code in text to do the same thing. So I think articles like this are useful to introduce everyone to various other paradigms, in case they might be applicable to a problem that you're working on. But it's never a "silver bullet" simply because each tool works best in certain problem domains. A friend tried to convince me that I should drink the functional programming koolaid, for instance, but on drilling deeper I found that he had never worked on a game, and didn't really grok what kinds of code you would normally end up putting in a game. What I'd love to see is an article that compared a lot of different processes and paradigms and talked about where each is most useful. Especially if that article surprised me. :) Problem is that most people are familiar with their own areas and toolsets; we'd probably need a team of people from different specialties to collaborate on it. Maybe a wiki where people could add in their experiences trying each approach or tool on their own problem domains? Then when someone does use functional programming to write a game, they could talk about the pros and cons and the kind of game they wrote. And that would be useful data.
- agentultra 12y ago> I think the types of problems each programmer tends to confront make up a large part of the disconnect between the ones who "drink the koolaid" of a programming paradigm or process and the people who are unimpressed. Sounds a lot like selection bias. Even in games there is a growing movement to stop the OOP paradigm and focus on data-oriented or "data flow". It's hard to know whether there are universally-applicable truths in a given paradigm... but it's rather nice to have the option. Flow-based tools are becoming popular in procedural content generation, have been in audio/music/dsp for some time, and are probably quite useful in other sort of "pipeline" analogies. But certainly past experience will play a large role in determining which tools you think are applicable to a given problem.
- SomeCallMeTim 12y agoSelection bias implies that people just like what they're familiar with; I'm sure that does keep people from switching to new tools, but that's not what I'm referring to. I'm talking about people who actually try new tools and reject them as useful to their problem domains; I try a lot of new tools, and some fit with the problems I solve better than others. I didn't quite remember what "data oriented meant," but a quick search brought up this article [1] on using data-driven design in games. It brings up some important points, but really the point of data flow design seems to be optimization and efficiency rather than making it easier to write the code. And I'm all for that, and I thank you for making me look that up since I will likely use that idea, but I'll keep my text editor for most of the game development I do going forward. Except maybe the particle systems, which as the article below points out, are typically already data oriented, and have had good data-flow-based tools for years. ;) [1] http://gamesfromwithin.com/data-oriented-design http://gamesfromwithin.com/data-oriented-design
- agentultra 12y agoI think you're referring to confirmation bias which is related but not quite the same. Never the less I understand what you're getting at. I took it to mean that the kinds of problems a programmer encounters is not sufficiently random to properly determine the usefulness of a given language or "paradigm." Which may keep people from switching to new, better tools (or from trying them out at all). The data-oriented design thing is interesting and something I've heard come and go over the years. Check out: http://www.dataorienteddesign.com/dodmain/ http://www.dataorienteddesign.com/dodmain/ and https://www.youtube.com/watch?v=ZHqFrNyLlpA https://www.youtube.com/watch?v=ZHqFrNyLlpA A video demonstration of Jai; a new compiler from Jonathan Blow designed around this paradigm. Cheers!
- robhawkes 12y ago"There are specific parts that could be reduced to flow based programming (FBP), but it seems like the wrong paradigm for writing an entire game, as an example close to me." Very much so! While I disagree slightly (a game could definitely be created this way – I have a background in game development), I'm 100% behind the statement that not everything has to be FBP. In my case, I'm using FBP for one specific part of my application that dynamically manipulates large quantities of data – FBP made sense. As for the rest of the application, FBP could work but it would be because I want to implement it in FBP rather than using FBP because it's better at solving the problem at hand. My intention with this article is to try and offer a fair view of one potential approach (out of many). I make a deliberate effort to try not to dictate what people should be using, rather leave it up to them to decide if it fits.
- valarauca1 12y agoOne of the main issues with Flow-Based Programming is often while it optimizes generally in the opposite direction of Functional-Programming (Currying vs. Topological Sorting I.E.: Execute when needed vs. Execute as soon as possible) is that you tend to lose control of your software. While I'm not suggesting a purely imperative approach, to a certain degree gains can be had in an imperative context over either Functional/Flow Programming via in depth knowledge of OS/Hardware processes and are generally NOT developer fast/friendly. From a personal stand point Flow-Based Programming is amazing for concurrency since it lets you see where you can use threads/processes. As which sections of code are less attached to the rest.
- falcolas 12y ago> it optimizes generally in the opposite direction of Functional-Programming Really? It strikes me as exactly the opposite - the output of each discrete component is solely dependent on its input. There is no shared state between components, only what is passed between them.
- valarauca1 12y agoYour addressing the platform specific solution not the global problem I am. If you build a compiler that uses Flow Based programming without dipping into imperative blocks (very possible, and they exist). You get the problems I describe.
- falcolas 12y agoFlow based programming actually has quite a bit in common with purely functional programming, in that the individual processes are generally stateless; their output depends solely on their input. The fact that the individual processes might be written in an imperative language simply doesn't matter at this point, so long as they always produce the same output for any given input. This is not true for all FBP processes obviously - there exist processes which read to and write from file handles, but much like the popular FP languages, these "tainted" processes are contained and separated from the remaining portions of the program via their connections to other processes. Most of the non-functional "programming" in Flow based programs actually occurs at the level where you are composing the connections between processes (be that the GUI, or in a hand-written method which makes the connections). One additional benefit of FBP which it shares with traditional FP programs is that they are capable of exhibiting "lazy" execution properties through the application of back pressure - by limiting how many packets of data a given connection can contain, you create a program which consumes only enough content from any given generator to keep the entire system occupied. Now then, can these abstractions break down if you start digging into the implementations of individual processes? Of course. It's not turtles all the way down, after all. Then again, all FP languages which run on Von Neumann based architectures suffer this problem. They all get compiled down to imperative machine code, and are all subject to the same state interruptions as every other program.
- zobzu 12y agoone advantage of this is that you can sandbox much smaller pieces of code.
- kornish 12y agoCould you clarify how visual programming helps with building smaller isolated projects over traditional text-based means?
- robhawkes 12y agoThere are a number of ways, the most obvious being that you can build that smaller, isolated project incredibly quickly using flow-based programming. The thing is though, if you find traditional approaches quicker and easier then they're probably the ones you should be using instead (I'd still recommend trying FBP).
- robhawkes 12y agoThat's what I'm finding, the ability to break things down into smaller "sandboxed" components can be incredibly powerful. It still comes with it's own pros and cons, though I've been really enjoying it so far.
- otakucode 12y agoVisual programming has interested me for a long time because it is amazing how many times it has been tried and the result been unusable. If I had the time and resources, I'd very much like to try developing a visual programming system that uses a user interface change that I think would greatly change things. Basically, using the middle mouse wheel to zoom in/out. If you've played the Supreme Commander game series, you will have the best idea of what I mean. As it is, you end up with absolutely everything trying to be jammed in to one gigantic flat space. That's terrible. Better would be zooming in on a 'box' (function, data structure, etc) and having its guts become the new 'context' you're working in. So far as I have been able to find, this has never been attempted.
- meemoo 12y agoWe are working with an SVG-based ZUI framework for noflo-ui/flowhub and have plans take the metaphor further: http://i.imgur.com/wQqvhlD.gifv http://i.imgur.com/wQqvhlD.gifv ZUI dataflow has been done in the past with Nuke. VFX people make some massively complex graphs with that tool.
- falcolas 12y agoA ZUI would be great for working with the FP paradigm, but good UIs are hard to make; I think it's one of the way that NoFlow is gaining the traction it is, the UI is pretty good.
- robhawkes 12y agoThe GUI with NoFlo is one of the major reasons why this is different to the previous attempts at FBP (that and it's JavaScript). What I also love is that it's all released under MIT: https://github.com/noflo https://github.com/noflo
- meemoo 12y ago> the UI is pretty good. Thank you, that is encouraging. We're working to make it really good.
- dragonwriter 12y ago
- aikah 12y ago"Flow" programming is great for non programmers.I used it to develop VST plugins on pc and when I was working in video editing(compositing). However,I wouldn't use that to develop web apps.It's often quicker to actually write the code,since app development is rarely just about piping stuff.
- robhawkes 12y agoYou'd be surprised how quickly you get used to the idea of connecting components together with pipes, even for Web applications. The additional benefit being that once you've written a component once you don't need to rewrite it ever again, or even copy and paste the code. The NoFlo site has some good examples: http://noflojs.org/example/ http://noflojs.org/example/
- petercooper 12y agoIn particular, I think the growth of Docker/containerization and microservices will make this an area worth keeping an eye on. I'm not convinced it makes sense at a micro level, but at the macro level of connecting larger services that have been coded in the traditional way.. there's something very appealing about the idea, especially in the way it could interact with ops' concerns too (like having a visual representation of scaling or moving services between machines).
- robhawkes 12y agoWeirdly, I hadn't really considered the macro level. My use-case has been incredibly granular (processing flows of large quantities of realtime data) but the idea of using it to monitor large-scale operations would be kinda cool. What I love is that it supports both uses by default!
- dharma1 12y agohttps://jujucharms.com/ https://jujucharms.com/ sounds pretty much like what you are describing
- 12y ago
- jamesfisher 12y agoHow is flow-based programming different to FRP?
- robhawkes 12y agoI don't know myself (not used FRP) but based on a quick look it seems to be based on asynchronous dataflow programming so I imagine there are a lot of similarities (seeing as FBP is based on dataflow too). I'd love to hear more from someone who has experience with both.
- jdmichal 12y agoThings like this are great for data-processing pipelines with an enumerated set of common operations, most of which input and output the same data type. For instance, any data that can be redefined as a matrix, since a lot of matrix operations result in another matrix (or a vector, which is just a restricted matrix). However, I can't help but feel that applying it any more generally is a hammer->nail issue. Or, at least, completely invalidates the idea of "massively reusable components". To make components more reusable, they have to make fewer assumptions about their input. To make fewer assumptions, they will typically need to do less. The reductio ad absurdum argument is that you end up with a bunch of components that each correspond to what would have been a line of code in a text editor, and all you've gained is a pretty picture.
- dharma1 12y agoThanks for the write up, has inspired me to give NoFlo a go. Reminds me of Reaktor (an app used to design audio DSP effects/instruments) Will be checking your blog for a more hands on series.
- robhawkes 12y agoGlad to hear it! I'm sure you'll enjoy trying out NoFlo, even if you don't end up using it for anything.
- falcolas 12y agoOne other thing I'd like to bring up - Go is a surprisingly strong tool for building flow based programs. It does have a few weaknesses, though, such as the lack of a graphical front end, but this isn't a requirement to write a flow based program. The other (more significant) weakness is that you will have to either deal with interface{} casting quite a bit, or write type specific versions of the basic utilities. On the plus side, you get virtually free concurrency and nearly purpose-built communication channels!
- bergie 12y agoYep, there is https://github.com/trustmaster/goflow https://github.com/trustmaster/goflow for that
- zeLaur 12y agoI have to admit, haven't read all comments... but: FBP, is nothing new ... In the enterprise world ( the world that I`m working in ) it is used quite frequently, and there are some products that live of this for example ( windows workflow foundation, workflowgen ) for the c# world. FBP is a way of doing things, let's say it is a pattern, like MVC ( less than more, but bare with me ) they each serve a specific task. To be a tad more clear, let's say that MVC is more of a generic on size fits all approach while the FBP is more of a custom tailored way of doing things for (highly) specific things. So, MVC would be great for a wiki and FBP would be great for the whole wiki item management from treating the adding, verifing, drafts, plublish etc etc. ( I`m sure there are a lot of better examples out there ...) The benefit of NoFlo is that they actually made something that is usually regarded as "enterprise-ish crap" and made it appealing. And again they did a great job with the visual editor, but from my own experience with visual flow design ... sooner that later you will be back to the code ( we you need to stray a bit from the Happy Path ). Also, I think that frameworks like NoFlow are a very good step for Node in general showing that a lot more interesting things could be done apart for the tasteless and colorless REST apis.