9 ms·

Pyflow – Visual and modular block programming in Python

229 points by amai 5y ago

dukeofdoom 5y ago
Recently Embaracedero (formerly Borland) released "Delphi for Python" allowing you use their visual tools with python too.
d--b 5y ago
Shameless plug: Same for Javascript and without the links :-) https://www.jigdev.com https://www.jigdev.com
howmayiannoyyou 5y ago
Can you imagine the popularity of a UI component driven version of this (along the lines of hypercard I guess), if web-based? I suspect the market for that would be MUCH bigger.
- riskneutral 5y ago
  Why would web-based ensure a much bigger market? If you have a one-click installation process, then a desktop app seems much more appropriate than a web page for a programming / data analysis tool.
  - ReleaseCandidat 5y ago
    Why not have both with a PWA :-D
- halflings 5y ago
  What do you mean by "UI component driven version of this" exactly? Like this, but to build UI components somehow? FWIW, there are plenty of graph-based dev tools, especially in the data science & ML world, and they seem to only fit the nerdiest of audiences (like mine!) ; not very practical for most people.
mrlinx 5y ago
How well does it handle web-requests to the outside?
dang 5y ago
Is this different from PyFlow – Visual scripting framework for Python – NodeRED alternative? - https://news.ycombinator.com/item?id=29955623 https://news.ycombinator.com/item?id=29955623 - Jan 2022 (53 comments) ?
- _joel 5y ago
  Looks like it. I thought I'd heard the name before...
- detaro 5y ago
  yes, appears to be an independent project, started after the one you linked petered out.
  - dang 5y ago
    Thanks!
- punnerud 5y ago
  Not the same. This post is to a git-repo with active development, compared to the first post where the last commit was more than two years ago. Both about PyFlow, but the first one should probably not be used. They also changed the license from Apache 2.0 to GPL-3.0 (probably a smart move). That make this post deserve some time on HN. Glad that someone is breathing fresh air into the project.
dbish 5y ago
Very unreal engine blueprint-like. I always wondered why that caught on with game engines but (partial) visual programming isn’t used by folks pretty much anywhere else besides some introductory programming courses/tutorials or kids projects. Personally, I think the abstractions could be useful for other domains like building/prototyping ML models too but so far no user demand for the tools.
- jefurii 5y ago
  There are at least two popular visual programming languages for electronic music: Max/MSP (cycling74.com) and PureData (puredata.info).
  - ReleaseCandidat 5y ago
    And Reaktor Core https://www.native-instruments.com/fileadmin/redaktion_upload/pdf/NI_REAKTOR5_Core_Manual_EN.pdf https://www.native-instruments.com/fileadmin/redaktion_uploa... and Primary https://www.native-instruments.com/fileadmin/ni_media/downloads/manuals/REAKTOR_6_Building_in_Primary_English_0419.pdf https://www.native-instruments.com/fileadmin/ni_media/downlo...
- andi999 5y ago
  Well there is labview with a dataflow model, multithreading use is extremely easy in such a setup.
- seemaze 5y ago
  This form of visual programming has been used in 3D modeling and visualization industries for 15+ years [0-3]. It's an excellent gateway to real programming, but any complexity quickly overwhelms the visual organization of nodes. [0]https://docs.blender.org/manual/en/2.79/render/blender_render/materials/nodes/introduction.html https://docs.blender.org/manual/en/2.79/render/blender_rende... [1]https://www.rhino3d.com/6/new/grasshopper/ https://www.rhino3d.com/6/new/grasshopper/ [2]https://blog.vectorworks.net/what-is-marionette-a-look-at-vectorworks-algorithmic-modeling-tool https://blog.vectorworks.net/what-is-marionette-a-look-at-ve... [3]https://dynamobim.org https://dynamobim.org
brrrrrm 5y ago
I like that this is basically Jupyter/Colab extended with an explicit cell dependence graph. However, I think this needs some serious brand work to distinguish it from "visual programming," which multiple commenters have already confused it for.
- throwaway17_17 5y ago
  Can you explain how this is not 'visual programming'? The interface seemingly allows for arbitrary code to be written and then graphically arranged in a dataflow-like format. If you mean that the fundamental elements of 'code' are the text inside each block, how is that fundamentally different than if each block was a black-box whose functionality was implemented using more blocks?
  - brrrrrm 5y ago
    The emphasis of this project seems to be text based coding and visual description of dependencies at the cell level. Visual programming, at least according to wikipedia, refers to "manipulating program elements graphically rather than by specifying them textually," which this project does not do.
    - throwaway17_17 5y ago
      Not really in disagreement. However, I think that there is more than one level of granularity to this project. If you were to step back to the level where someone has a 'library' of these blocks of code (which seem to be able to present a 'black-box' style presentation) and then visually placing them and connecting the inputs and outputs of those various blocks is essentially visual programming within the meaning of the standard definition, like from Wikipedia.
    - zozbot234 5y ago
      Dependency relations are very much "program elements".
    - dekhn 5y ago
      The graphical elements themselves contain code (all visual programming systems work this way, even if the code is hidden). That's how new program elements are born. Visual workflow languages work in a bunch of different ways. Nodes, in Blender, is a visual programming environment. Blockly is a visual programming environment. Those nodes in scratch have code behind them!
agumonkey 5y ago
It's surprising how the lazy node graph thing is spreading. I find it a superb model for most things (and, broken record moment: sidefx houdini or alias maya can show you how far one can go in this model. hint: james webb far).
- ReleaseCandidat 5y ago
  And 3D Compositing software like Nuke, Flame, Fusion, .... And don't forget about Blender!
  - agumonkey 5y ago
    yeah all these compositors are from the same era.. it's funny how they all rapidly and naturally converged onto these ergonomics blender is closing in
- throw10920 5y ago
  Where does the "lazy" come from? Is that just another way to say that it's a dataflow graph?
  - agumonkey 5y ago
    yes, lazy, dataflow, reactive, I wrap all this in the same bag. Am I wrong ?
    - chrisseaton 5y ago
      Lazy is usually used to mean demand-driven, as opposed to input-driven, in my experience, so exactly the opposite of this, but different people mean different things. (Did a lot of my PhD on dataflow.)
      - agumonkey 5y ago
        These systems are a bit of both though. In compositing, unless you add a final "view" node, IIRC, node computations are not triggered.
tda 5y ago
I am wondering if this could work for async programming. I once used a Discrete Event Simulation software that had an excellen graph based program editor that really showed the dependencies of the model very clearly. I later realized that Discrete Event Simulation is conceptually very similar to async programming. I would love to explore async programs as an animated graph. Anyone know if such a thing has been tried? The modelling software I was thinking about is FlexSim by the way
- idealmedtech 5y ago
  LabVIEW uses a graphical language and gives very easy access to parallelism and message-based software architectures, though it can be quite difficult to learn and is plagued by lots of bad existing code. See OpenG [1] for some examples of good code. [1] - https://www.ni.com/en-us/support/downloads/tools-network/download.openg-libraries-for-labview.html#379041 https://www.ni.com/en-us/support/downloads/tools-network/dow...
tylerl 5y ago
Does anyone know of a more generalized framework for doing this kind of thing? I'd been meaning to write a framework kind of like this for some time, but never got around to it, and was hoping someone else would. This one unfortunately doesn't really check the important boxes, but it's a good start. I was hoping more for: * Target language agnostic (this one seems to get mostly there) -- the nodes communicate data/logic flow, you then serialize/deserialize accordingly. * Focus on data flow, not just execution -- IO from nodes, visual indicators of data types (colors or something) * Capable of visually encapsulating complexity -- define flows within flows * Ideally embeddable in web apps (e.g a browser/electron frontend or something) These are pretty popular to embed in complex "design" oriented applications, especially ones that involve crafting procedures by non-programmers (e.g. artists, data scientists, etc). Examples where this is implemented that come to mind include Blender, Unity, and Unreal. A core part of the fundamental design of each one of the successful implementations is that they allow efficient code to be crafted by people who don't think they understand code. Making it visual helps engage the brains of certain kinds of people. The "code as text" paradigm is spatially efficient, but it's like a brick wall for some people.
- epilys 5y ago
  There's Google's Blockly: https://developers.google.com/blockly https://developers.google.com/blockly
- solarkraft 5y ago
  Hey, same about looking for a generalized framework, but I'd approach it from the other side. I'm pretty okay with the general shape of code (a nested tree-type structure), but think the possible interactions are unnecessarily awkward by being forced into a text editor. You ought to be able to effortlessly fold, move around and disable/enable blocks of code. There's not much of a point in allowing indentation or whitespace mistakes and it doesn't usually make much sense to select a part of a token (or often even just a token without the block it's attached to). These issues can mostly be fixed by representing tokens and lines in a tree-like structure of nodes, for which useful editors already exist. IDEs try to retrofit these things into their text model (the most advanced one I'm aware of being IntelliJ), with automatic indentation fixing and folding, but even the best attempts aren't even using 20% of the potential. My most jarring example of how bad IDEs still are at this is folding + disabling (commenting out): When I comment out a block of code because I don't care about it that is folded because I don't care about it, it shows it back to me even though I don't want to see it because I double-don't care about it! (Side note: I'm aware XCode doesn't do this, but it's far from general) There is so much effort put into concepts, parsing and whatever, and everything uses trees, because that's the natural way of reasoning about code. Why are we still stuck interfacing with it through an awkwardly serialized version?
raintrees 5y ago
As a past user of Borland's ObjectVision, I would appreciate seeing this project developed far further... Similar to the Mindstorms programming interface, sometimes the object-based modular containment assists with abstraction/black box thinking.
phailhaus 5y ago
> Link blocks to highlight dependencies, Pyflow will then automatically run your blocks in the correct order This is a problem. If you require users to define dependencies for you, they will make mistakes and things will go wrong in ways that are incredibly difficult to debug. This is already a huge pain point with Jupyter notebooks, and they're only one-dimensional! Have you considered going the route of Observable notebooks [1]? In an Observable notebook, each cell can export at most one variable. This means that the notebook can automatically determine dependencies statically, run cells in the correct order, and even show the user a viz of dependencies. [1] https://observablehq.com/ https://observablehq.com/
- brrrrrm 5y ago
  Isn't this readily solved by a user convention such as only updating one variable each cell? Why limit users based on mistakes they haven't made yet?
  - phailhaus 5y ago
    User conventions don't work, because there is no mechanism for enforcement other than "hope they remember!" I would say most users don't even know about this convention, given I work in an environment where notebooks are used heavily and have never heard of this. Because these mistakes are allowed to happen silently, they will happen, and it will almost certainly make the code harder to manage. On top of that, Pyflow seems to rely on the user to define dependencies between blocks as well. This means that the user would have to both perfectly maintain the convention of one-variable-per-block, but also link every block that touches that variable going forward. Without ever making a mistake. It's such a well-defined graph problem that it's practically made for computers to handle, not fleshy people. This is why tools are so important: they can impose constraints that allow you to act more freely elsewhere. When I change a cell in Observable or add a reference, I don't have to think. The engine handles dependencies for me, without fail. I can't make an entire class of errors, which means I can just focus on functionality.
- mrtranscendence 5y ago
  Wait, what is it about exporting only one variable at a time that makes it possible to determine dependencies statically? Why not two variables? Or three? If you're determining dependencies based on variable names it seems like that would be possible.
spywaregorilla 5y ago
I love blueprints in ue4, but I think I don't like the look of this. It's weird to me that each node seems to be an arbitrary block of code. I've long wished for such a feature in unreal's blueprint visual programming language. However, I wouldn't want it to be the default. That kind of undermines the point of it all. I would want 99% of nodes to be single function calls.
- lgessler 5y ago
  It makes more sense when you consider that this seems to be intended as an alternative to (Jupyter, etc.) notebooks, which are a major way of communicating methods in the data science segment of the Python community. Notebooks are linear sequences of cells with an arbitrary number of lines of code inside of them, and these cells can be executed one at a time in arbitrary order. A lot of the time the dependencies between these cells (whether it be conceptually, programmatically, etc.) is not linear. There therefore arises a lot of cognitive load in both trying to present (via comments, etc.) this structure as a writer and parse this structure as a reader. I imagine that is the kind of problem this is meant to solve--drawing out and visually solidifying that structure. From this perspective, I'm not sure if there's much apparent value in making the 1 cell <==> 1 function call constraint the default.
tekacs 5y ago
Another attempt in this space recently is https://natto.dev https://natto.dev, by Paul Shen (https://twitter.com/_paulshen https://twitter.com/_paulshen). It's a really interesting platform and his Twitter feed shows you all sorts of interesting and innovative uses he's making of it so far! See this previous HN thread on the topic (though it's evolved quite a bit since!) -- https://news.ycombinator.com/item?id=26790285 https://news.ycombinator.com/item?id=26790285
- mrits 5y ago
  I worked on something very similar for my company about 10 years ago. The lesson I learned is it was much more robust to have a flow of data than a flow of code. I still think it's really cool though.
TruthWillHurt 5y ago
Cool. Now the question is where does it run, and can we split blocks to serverless functions, or have a "cluster-mode" where each block can be executed on dedicated/specialized/available node.
- TruthWillHurt 5y ago
  Also, would be great to have generic condition & state functions, so the flow can define a state-machine.
realitysballs 5y ago
It’s like the grasshopper of Python
architectonic 5y ago
I had worked on such a tool by extending pyqtgraph's flowcharts. Mainly executing the nodes when their input changes and showing a glowing box around the one running at the moment. Flowcharts supports multiple terminals already and I had added some custom nodes. Node execution must be delegated to a background thread so the gui is not blocked while the node executes. Doubleclicking a node shows a window for setting up the node. A custom class of nodes shows upon doubleclicking a simple python editor, upon execution the terminals get automatically the values of the variables carrying the same name as the terminals. It has been a lot of fun. I had used petl and pyspark for lazy loading so one could build large etl workflows.
erwincoumans 5y ago
It looks great. I've been using ImGUI + ImGUI Node Editor for a synthesizer GUI. It should be easy to create Python bindings for that as well. In case someone is interested in such Python bindings, reply or +1. See https://github.com/thedmd/imgui-node-editor https://github.com/thedmd/imgui-node-editor and my simple synth GUI here: https://www.youtube.com/watch?v=LPwPJuKbASs https://www.youtube.com/watch?v=LPwPJuKbASs
sdepablos 5y ago
How it this different from any DAG-centered orchestration tool like Dagster or Prefect?
throwamon 5y ago
This is very close to what I wish ObservableHQ/Pluto.jl had. Their reactive model already allows you to branch off, connect arbitrary nodes and re-run only what's needed, but the visual ordering of the cells is still very inflexible.
oofbey 5y ago
Ugh. Yet another visual ML programming tool. They look great for demos. And are a complete disaster when you try to do real work. At any real-world level of complexity, managing boxes and arrows is totally untenable, and you need code. All the world's coding tools are great for code. Doing git diffs or code reviews on boxes and arrows just doesn't work. This will join a crowded graveyard of visual ML tools along with the likes of ML.Net or Azure ML Studio Whatever, KNIME, CreateML, PyCaret, Teachable Machine, etc etc etc. The pattern is very predictable. Nice demo. Excitement hits real-world problem. Complexity. Implosion.
- Jeff_Brown 5y ago
  It looks like the graphical part is optional. You could just have one big node that was a Python program. Hopefully that means you could use the arrows and boxes only where they are natural. They'll at least be natural to impose at the top level of your program. They might also be useful at lower levels, if it's possible to make a functional abstraction out of a signal diagram. Diffs will still be a problem though. I wouldn't touch this stuff with a ten foot pole.
- disconcision 5y ago
  > you need code. there's code. did you click through?
- eyeball 5y ago
  pycaret is not really a visual tool is it?
- whoevercares 5y ago
  Citizen data scientists are real in big enterprise, and now comes citizen “developers”. Low-code / no-code is big for a reason
whatisnotreal 5y ago
this is dumb af. also nice work are you hiring
dekhn 5y ago
Excellent- QGraphicsView for the win. BTW, if anybody gets WebAssemly with PyQt5 working, such that I can made a web page that is QGraphicsView, I would be sooooo happy.
- jcelerier 5y ago
  It's not PyQt but QtWidgets in C++ directly - https://ossia.io/score-web https://ossia.io/score-web uses QGraphicsView for the central view. It's not been updated to Qt 6 which fixes a lot of drag'n'drop-related bugs yet so the experience is icky.
  - dekhn 5y ago
    loading that page immediately fails with an error for me (uptodate Chrome). I'll be back in 5 years to ask the same question.
    - jcelerier 5y ago
      Huh, on which platform ? I tested it on Windows, Linux, with Chrome and Firefox ; it even loads on my android phone even if not particularly useful.
      - dekhn 5y ago
        Windows 10. Tried chrome on Mac and it was fine. Do you know if the entire build process for the site is available (IE, a simple example I can steal to get a QGraphicsView (C++) in a web page?
        jcelerier 5y ago
        I pretty much just use the emscripten CMake toolchain with my Qt app (of course a WASM build of Qt is needed but TQTC provides some I believe ?)
kumarvvr 5y ago
I am building a web application (FastAPI+PostgreSQL+Angular) that includes some user scripting. Any suggestions for libraries that do this sort of thing, but have an option to be integrated into existing apps?
- quinnjh 5y ago
  Node red?
otagekki 5y ago
My colleagues wrote PyF, using the same idea of code written in boxes that you link together. It saved all code blocks in an XML file. We used it on financial data. One of our programs was so complex that we fell back on using a text editor to directly edit the XML instead of using the GUI. The project was published on pypi but for some reason the source code was never published and it is in the same state since 2011, using python 2 and all. I left the place 2 years ago but my colleagues who still work there said it's still mission-critical. A kind of Cobol.
d4rkp4ttern 5y ago
Reminds me of coding Alfred workflows in MacOS

229 points by amai 5y ago

Recently Embaracedero (formerly Borland) released "Delphi for Python" allowing you use their visual tools with python too.

d--b 5y ago

Shameless plug: Same for Javascript and without the links :-) https://www.jigdev.com https://www.jigdev.com

howmayiannoyyou 5y ago

Can you imagine the popularity of a UI component driven version of this (along the lines of hypercard I guess), if web-based? I suspect the market for that would be MUCH bigger.

riskneutral 5y ago
Why would web-based ensure a much bigger market? If you have a one-click installation process, then a desktop app seems much more appropriate than a web page for a programming / data analysis tool.
- ReleaseCandidat 5y ago
  Why not have both with a PWA :-D
halflings 5y ago
What do you mean by "UI component driven version of this" exactly? Like this, but to build UI components somehow? FWIW, there are plenty of graph-based dev tools, especially in the data science & ML world, and they seem to only fit the nerdiest of audiences (like mine!) ; not very practical for most people.

mrlinx 5y ago

How well does it handle web-requests to the outside?

dang 5y ago

Is this different from PyFlow – Visual scripting framework for Python – NodeRED alternative? - https://news.ycombinator.com/item?id=29955623 https://news.ycombinator.com/item?id=29955623 - Jan 2022 (53 comments) ?

_joel 5y ago
Looks like it. I thought I'd heard the name before...
detaro 5y ago
yes, appears to be an independent project, started after the one you linked petered out.
- dang 5y ago
  Thanks!
punnerud 5y ago
Not the same. This post is to a git-repo with active development, compared to the first post where the last commit was more than two years ago. Both about PyFlow, but the first one should probably not be used. They also changed the license from Apache 2.0 to GPL-3.0 (probably a smart move). That make this post deserve some time on HN. Glad that someone is breathing fresh air into the project.

dbish 5y ago

Very unreal engine blueprint-like. I always wondered why that caught on with game engines but (partial) visual programming isn’t used by folks pretty much anywhere else besides some introductory programming courses/tutorials or kids projects. Personally, I think the abstractions could be useful for other domains like building/prototyping ML models too but so far no user demand for the tools.

jefurii 5y ago
There are at least two popular visual programming languages for electronic music: Max/MSP (cycling74.com) and PureData (puredata.info).
- ReleaseCandidat 5y ago
  And Reaktor Core https://www.native-instruments.com/fileadmin/redaktion_upload/pdf/NI_REAKTOR5_Core_Manual_EN.pdf https://www.native-instruments.com/fileadmin/redaktion_uploa... and Primary https://www.native-instruments.com/fileadmin/ni_media/downloads/manuals/REAKTOR_6_Building_in_Primary_English_0419.pdf https://www.native-instruments.com/fileadmin/ni_media/downlo...
andi999 5y ago
Well there is labview with a dataflow model, multithreading use is extremely easy in such a setup.
seemaze 5y ago
This form of visual programming has been used in 3D modeling and visualization industries for 15+ years [0-3]. It's an excellent gateway to real programming, but any complexity quickly overwhelms the visual organization of nodes. [0]https://docs.blender.org/manual/en/2.79/render/blender_render/materials/nodes/introduction.html https://docs.blender.org/manual/en/2.79/render/blender_rende... [1]https://www.rhino3d.com/6/new/grasshopper/ https://www.rhino3d.com/6/new/grasshopper/ [2]https://blog.vectorworks.net/what-is-marionette-a-look-at-vectorworks-algorithmic-modeling-tool https://blog.vectorworks.net/what-is-marionette-a-look-at-ve... [3]https://dynamobim.org https://dynamobim.org

brrrrrm 5y ago

I like that this is basically Jupyter/Colab extended with an explicit cell dependence graph. However, I think this needs some serious brand work to distinguish it from "visual programming," which multiple commenters have already confused it for.

throwaway17_17 5y ago
Can you explain how this is not 'visual programming'? The interface seemingly allows for arbitrary code to be written and then graphically arranged in a dataflow-like format. If you mean that the fundamental elements of 'code' are the text inside each block, how is that fundamentally different than if each block was a black-box whose functionality was implemented using more blocks?
- brrrrrm 5y ago
  The emphasis of this project seems to be text based coding and visual description of dependencies at the cell level. Visual programming, at least according to wikipedia, refers to "manipulating program elements graphically rather than by specifying them textually," which this project does not do.
  - throwaway17_17 5y ago
    Not really in disagreement. However, I think that there is more than one level of granularity to this project. If you were to step back to the level where someone has a 'library' of these blocks of code (which seem to be able to present a 'black-box' style presentation) and then visually placing them and connecting the inputs and outputs of those various blocks is essentially visual programming within the meaning of the standard definition, like from Wikipedia.
  - zozbot234 5y ago
    Dependency relations are very much "program elements".
  - dekhn 5y ago
    The graphical elements themselves contain code (all visual programming systems work this way, even if the code is hidden). That's how new program elements are born. Visual workflow languages work in a bunch of different ways. Nodes, in Blender, is a visual programming environment. Blockly is a visual programming environment. Those nodes in scratch have code behind them!

agumonkey 5y ago

It's surprising how the lazy node graph thing is spreading. I find it a superb model for most things (and, broken record moment: sidefx houdini or alias maya can show you how far one can go in this model. hint: james webb far).

ReleaseCandidat 5y ago
And 3D Compositing software like Nuke, Flame, Fusion, .... And don't forget about Blender!
- agumonkey 5y ago
  yeah all these compositors are from the same era.. it's funny how they all rapidly and naturally converged onto these ergonomics blender is closing in
throw10920 5y ago
Where does the "lazy" come from? Is that just another way to say that it's a dataflow graph?
- agumonkey 5y ago
  yes, lazy, dataflow, reactive, I wrap all this in the same bag. Am I wrong ?
  - chrisseaton 5y ago
    Lazy is usually used to mean demand-driven, as opposed to input-driven, in my experience, so exactly the opposite of this, but different people mean different things. (Did a lot of my PhD on dataflow.)
    - agumonkey 5y ago
      These systems are a bit of both though. In compositing, unless you add a final "view" node, IIRC, node computations are not triggered.

tda 5y ago

I am wondering if this could work for async programming. I once used a Discrete Event Simulation software that had an excellen graph based program editor that really showed the dependencies of the model very clearly. I later realized that Discrete Event Simulation is conceptually very similar to async programming. I would love to explore async programs as an animated graph. Anyone know if such a thing has been tried? The modelling software I was thinking about is FlexSim by the way

idealmedtech 5y ago
LabVIEW uses a graphical language and gives very easy access to parallelism and message-based software architectures, though it can be quite difficult to learn and is plagued by lots of bad existing code. See OpenG [1] for some examples of good code. [1] - https://www.ni.com/en-us/support/downloads/tools-network/download.openg-libraries-for-labview.html#379041 https://www.ni.com/en-us/support/downloads/tools-network/dow...

tylerl 5y ago

Does anyone know of a more generalized framework for doing this kind of thing? I'd been meaning to write a framework kind of like this for some time, but never got around to it, and was hoping someone else would. This one unfortunately doesn't really check the important boxes, but it's a good start. I was hoping more for: * Target language agnostic (this one seems to get mostly there) -- the nodes communicate data/logic flow, you then serialize/deserialize accordingly. * Focus on data flow, not just execution -- IO from nodes, visual indicators of data types (colors or something) * Capable of visually encapsulating complexity -- define flows within flows * Ideally embeddable in web apps (e.g a browser/electron frontend or something) These are pretty popular to embed in complex "design" oriented applications, especially ones that involve crafting procedures by non-programmers (e.g. artists, data scientists, etc). Examples where this is implemented that come to mind include Blender, Unity, and Unreal. A core part of the fundamental design of each one of the successful implementations is that they allow efficient code to be crafted by people who don't think they understand code. Making it visual helps engage the brains of certain kinds of people. The "code as text" paradigm is spatially efficient, but it's like a brick wall for some people.

epilys 5y ago
There's Google's Blockly: https://developers.google.com/blockly https://developers.google.com/blockly
solarkraft 5y ago
Hey, same about looking for a generalized framework, but I'd approach it from the other side. I'm pretty okay with the general shape of code (a nested tree-type structure), but think the possible interactions are unnecessarily awkward by being forced into a text editor. You ought to be able to effortlessly fold, move around and disable/enable blocks of code. There's not much of a point in allowing indentation or whitespace mistakes and it doesn't usually make much sense to select a part of a token (or often even just a token without the block it's attached to). These issues can mostly be fixed by representing tokens and lines in a tree-like structure of nodes, for which useful editors already exist. IDEs try to retrofit these things into their text model (the most advanced one I'm aware of being IntelliJ), with automatic indentation fixing and folding, but even the best attempts aren't even using 20% of the potential. My most jarring example of how bad IDEs still are at this is folding + disabling (commenting out): When I comment out a block of code because I don't care about it that is folded because I don't care about it, it shows it back to me even though I don't want to see it because I double-don't care about it! (Side note: I'm aware XCode doesn't do this, but it's far from general) There is so much effort put into concepts, parsing and whatever, and everything uses trees, because that's the natural way of reasoning about code. Why are we still stuck interfacing with it through an awkwardly serialized version?

raintrees 5y ago

As a past user of Borland's ObjectVision, I would appreciate seeing this project developed far further... Similar to the Mindstorms programming interface, sometimes the object-based modular containment assists with abstraction/black box thinking.

phailhaus 5y ago

> Link blocks to highlight dependencies, Pyflow will then automatically run your blocks in the correct order This is a problem. If you require users to define dependencies for you, they will make mistakes and things will go wrong in ways that are incredibly difficult to debug. This is already a huge pain point with Jupyter notebooks, and they're only one-dimensional! Have you considered going the route of Observable notebooks [1]? In an Observable notebook, each cell can export at most one variable. This means that the notebook can automatically determine dependencies statically, run cells in the correct order, and even show the user a viz of dependencies. [1] https://observablehq.com/ https://observablehq.com/

brrrrrm 5y ago
Isn't this readily solved by a user convention such as only updating one variable each cell? Why limit users based on mistakes they haven't made yet?
- phailhaus 5y ago
  User conventions don't work, because there is no mechanism for enforcement other than "hope they remember!" I would say most users don't even know about this convention, given I work in an environment where notebooks are used heavily and have never heard of this. Because these mistakes are allowed to happen silently, they will happen, and it will almost certainly make the code harder to manage. On top of that, Pyflow seems to rely on the user to define dependencies between blocks as well. This means that the user would have to both perfectly maintain the convention of one-variable-per-block, but also link every block that touches that variable going forward. Without ever making a mistake. It's such a well-defined graph problem that it's practically made for computers to handle, not fleshy people. This is why tools are so important: they can impose constraints that allow you to act more freely elsewhere. When I change a cell in Observable or add a reference, I don't have to think. The engine handles dependencies for me, without fail. I can't make an entire class of errors, which means I can just focus on functionality.
mrtranscendence 5y ago
Wait, what is it about exporting only one variable at a time that makes it possible to determine dependencies statically? Why not two variables? Or three? If you're determining dependencies based on variable names it seems like that would be possible.

spywaregorilla 5y ago

I love blueprints in ue4, but I think I don't like the look of this. It's weird to me that each node seems to be an arbitrary block of code. I've long wished for such a feature in unreal's blueprint visual programming language. However, I wouldn't want it to be the default. That kind of undermines the point of it all. I would want 99% of nodes to be single function calls.

lgessler 5y ago
It makes more sense when you consider that this seems to be intended as an alternative to (Jupyter, etc.) notebooks, which are a major way of communicating methods in the data science segment of the Python community. Notebooks are linear sequences of cells with an arbitrary number of lines of code inside of them, and these cells can be executed one at a time in arbitrary order. A lot of the time the dependencies between these cells (whether it be conceptually, programmatically, etc.) is not linear. There therefore arises a lot of cognitive load in both trying to present (via comments, etc.) this structure as a writer and parse this structure as a reader. I imagine that is the kind of problem this is meant to solve--drawing out and visually solidifying that structure. From this perspective, I'm not sure if there's much apparent value in making the 1 cell <==> 1 function call constraint the default.

tekacs 5y ago

Another attempt in this space recently is https://natto.dev https://natto.dev, by Paul Shen (https://twitter.com/_paulshen https://twitter.com/_paulshen). It's a really interesting platform and his Twitter feed shows you all sorts of interesting and innovative uses he's making of it so far! See this previous HN thread on the topic (though it's evolved quite a bit since!) -- https://news.ycombinator.com/item?id=26790285 https://news.ycombinator.com/item?id=26790285

mrits 5y ago
I worked on something very similar for my company about 10 years ago. The lesson I learned is it was much more robust to have a flow of data than a flow of code. I still think it's really cool though.

TruthWillHurt 5y ago

Cool. Now the question is where does it run, and can we split blocks to serverless functions, or have a "cluster-mode" where each block can be executed on dedicated/specialized/available node.

TruthWillHurt 5y ago
Also, would be great to have generic condition & state functions, so the flow can define a state-machine.

realitysballs 5y ago

It’s like the grasshopper of Python

architectonic 5y ago

I had worked on such a tool by extending pyqtgraph's flowcharts. Mainly executing the nodes when their input changes and showing a glowing box around the one running at the moment. Flowcharts supports multiple terminals already and I had added some custom nodes. Node execution must be delegated to a background thread so the gui is not blocked while the node executes. Doubleclicking a node shows a window for setting up the node. A custom class of nodes shows upon doubleclicking a simple python editor, upon execution the terminals get automatically the values of the variables carrying the same name as the terminals. It has been a lot of fun. I had used petl and pyspark for lazy loading so one could build large etl workflows.

erwincoumans 5y ago

It looks great. I've been using ImGUI + ImGUI Node Editor for a synthesizer GUI. It should be easy to create Python bindings for that as well. In case someone is interested in such Python bindings, reply or +1. See https://github.com/thedmd/imgui-node-editor https://github.com/thedmd/imgui-node-editor and my simple synth GUI here: https://www.youtube.com/watch?v=LPwPJuKbASs https://www.youtube.com/watch?v=LPwPJuKbASs

sdepablos 5y ago

How it this different from any DAG-centered orchestration tool like Dagster or Prefect?

throwamon 5y ago

This is very close to what I wish ObservableHQ/Pluto.jl had. Their reactive model already allows you to branch off, connect arbitrary nodes and re-run only what's needed, but the visual ordering of the cells is still very inflexible.

oofbey 5y ago

Ugh. Yet another visual ML programming tool. They look great for demos. And are a complete disaster when you try to do real work. At any real-world level of complexity, managing boxes and arrows is totally untenable, and you need code. All the world's coding tools are great for code. Doing git diffs or code reviews on boxes and arrows just doesn't work. This will join a crowded graveyard of visual ML tools along with the likes of ML.Net or Azure ML Studio Whatever, KNIME, CreateML, PyCaret, Teachable Machine, etc etc etc. The pattern is very predictable. Nice demo. Excitement hits real-world problem. Complexity. Implosion.

Jeff_Brown 5y ago
It looks like the graphical part is optional. You could just have one big node that was a Python program. Hopefully that means you could use the arrows and boxes only where they are natural. They'll at least be natural to impose at the top level of your program. They might also be useful at lower levels, if it's possible to make a functional abstraction out of a signal diagram. Diffs will still be a problem though. I wouldn't touch this stuff with a ten foot pole.
disconcision 5y ago
> you need code. there's code. did you click through?
eyeball 5y ago
pycaret is not really a visual tool is it?
whoevercares 5y ago
Citizen data scientists are real in big enterprise, and now comes citizen “developers”. Low-code / no-code is big for a reason

whatisnotreal 5y ago

this is dumb af. also nice work are you hiring

dekhn 5y ago

Excellent- QGraphicsView for the win. BTW, if anybody gets WebAssemly with PyQt5 working, such that I can made a web page that is QGraphicsView, I would be sooooo happy.

jcelerier 5y ago
It's not PyQt but QtWidgets in C++ directly - https://ossia.io/score-web https://ossia.io/score-web uses QGraphicsView for the central view. It's not been updated to Qt 6 which fixes a lot of drag'n'drop-related bugs yet so the experience is icky.
- dekhn 5y ago
  loading that page immediately fails with an error for me (uptodate Chrome). I'll be back in 5 years to ask the same question.
  - jcelerier 5y ago
    Huh, on which platform ? I tested it on Windows, Linux, with Chrome and Firefox ; it even loads on my android phone even if not particularly useful.
    - dekhn 5y ago
      Windows 10. Tried chrome on Mac and it was fine. Do you know if the entire build process for the site is available (IE, a simple example I can steal to get a QGraphicsView (C++) in a web page?
      - jcelerier 5y ago
        I pretty much just use the emscripten CMake toolchain with my Qt app (of course a WASM build of Qt is needed but TQTC provides some I believe ?)

kumarvvr 5y ago

I am building a web application (FastAPI+PostgreSQL+Angular) that includes some user scripting. Any suggestions for libraries that do this sort of thing, but have an option to be integrated into existing apps?

quinnjh 5y ago
Node red?

otagekki 5y ago

My colleagues wrote PyF, using the same idea of code written in boxes that you link together. It saved all code blocks in an XML file. We used it on financial data. One of our programs was so complex that we fell back on using a text editor to directly edit the XML instead of using the GUI. The project was published on pypi but for some reason the source code was never published and it is in the same state since 2011, using python 2 and all. I left the place 2 years ago but my colleagues who still work there said it's still mission-critical. A kind of Cobol.

d4rkp4ttern 5y ago

Reminds me of coding Alfred workflows in MacOS