6 ms·

Rete – JavaScript framework for node-based visual programming

134 points by Immortalin 8y ago

gedy 8y ago
Really nice, haven't figured out what it's capable of, but reminds of a similar interface I wrote from scratch about 15 year ago in Java Swing. Blows me away how powerful browsers and JS have become. Really amazing that you can do this with so few lines of code, compared the (tens of?) thousands lines I wrote to do similar.
madrox 8y ago
Any time I see something like this I'm reminded of Yahoo Pipes and how ahead of its time it was. Is there anything like it that's designed for enterprise ETL these? It occurs to me it's a much better way of thinking about data flow problems than code.
- oldmanhorton 8y ago
  I think https://flow.microsoft.com/en-us/ https://flow.microsoft.com/en-us/ intends to be an enterprise version of this idea, but I've never used it.
  - ptrott2017 8y ago
    for enterprise ETL - there is also kettle which has been around for a long time- which has a desktop and web authoring client. https://github.com/pentaho/pentaho-kettle https://github.com/pentaho/pentaho-kettle
- agumonkey 8y ago
  Node based / dataflow UI are also the bread and butter of CGI and compositing industry. It's hyper efficient both for machine because it's often freely memoized/cached and user (think of it as a simplified haskell at your fingertips) because the stateless and unlimited-undo aspects. I'm still pretty sure it should be the default UI for half computer introduction. Text for the rest, with the right paradigm of course.
  - codetrotter 8y ago
    Node based user interfaces can be great indeed. Unreal Engine 4 has a visual scripting system called Blueprints that is very impressive. https://docs.unrealengine.com/en-US/Engine/Blueprints https://docs.unrealengine.com/en-US/Engine/Blueprints Another example of a node based user interface can be found in the open source 3d modeling and animation software Blender. https://docs.blender.org/manual/en/dev/render/blender_render/materials/nodes/introduction.html https://docs.blender.org/manual/en/dev/render/blender_render... The open source virtual modular synthesizer VCV Rack is node based as well, though it is so “by nature” in the sense that hardware modular synthesizer work in fashion of connecting inputs and outputs with patch cables. https://www.vcvrack.com/ https://www.vcvrack.com/ When done right, as all of the three I mentioned above in my opinion do, node based user interfaces are great. Not all node based user interfaces are equally good. I used LabView a bit at a course at the university and did not enjoy their system. The LabView node based user interface felt very clunky the whole time, even for doing simple things. I’ve been wondering occasionally whether certain kinds of data processing that I am doing with Unix tools could benefit from a graphical system. It’s not that I am not comfortable with the command line, but even as a rather die-hard user of Linux and FreeBSD I must admit that the command line has several shortcomings. Of course, designing a graphical successor to Unix pipelines is a major undertaking that will require through thought. But I think that keeping that idea in the back of ones mind while implementing some smaller node based systems could eventually lead to realization of most if not all of the considerations that must be kept in mind for the design of a graphical successor to Unix pipelines.
- bpicolo 8y ago
  Domo[0]. Their visual ETL is pretty good. https://www.domo.com/ https://www.domo.com/
- monstrado 8y ago
  Two come to mind. Apache NiFi: http://nifi.apache.org/ http://nifi.apache.org/ StreamSets: https://streamsets.com/opensource/ https://streamsets.com/opensource/ Side note, I'm an Apache NiFi committer and have had great success using it for enterprise ETL in a previous role.
  - tmikaeld 8y ago
    Seems likestreamsets don't include a visual designer in the open source core? And no price, which seems standard with ETLs https://streamsets.com/subscriptions https://streamsets.com/subscriptions
- pjmlp 8y ago
  Yes, Bizztalk would be one of them.
- statictype 8y ago
  We have one that's a little more complex than Yahoo Pipes. Small Demo: https://s3.amazonaws.com/ecyber.public/lucy-use-cases/1/video.mp4 https://s3.amazonaws.com/ecyber.public/lucy-use-cases/1/vide... http://lucy.iviva.com http://lucy.iviva.com
feiss 8y ago
This is nice, great job! I started one framework very similar a couple of years ago, but never finished. I'm going to take a good look at this :)
c-rack 8y ago
The name reminds me on the Rete algorithm, which is pretty cool, but unrelated: https://en.wikipedia.org/wiki/Rete_algorithm https://en.wikipedia.org/wiki/Rete_algorithm
- hodgesrm 8y ago
  I found the name confusing for exactly this reason. Rete is a big deal in rule-based systems.
  - kjhughes 8y ago
    Agreed. It's such a unique name for a classic algorithm and is so thoroughly established in symbolic pattern matching for production systems that I'd seriously recommend against adopting it for any other purpose in CS.
Kranar 8y ago
Is there a way to represent recursive nodes or recursion in general?
- Ni55aN 8y ago
  It can be represented visually, but the engine does not support processing recursive nodes
outofpaper 8y ago
Node Red is an other node based visual programming tool that is inspired by Yahoo pipes.
zoggenhoff 8y ago
Looks good. Time to brush up on my websockets and glue this into something.
freen 8y ago
The recurring problem I've found with node-based visual programming is that it's very challenging to work with multiple layers of abstraction and that refactoring can be incredibly challenging.
- otakucode 8y ago
  Those are definitely two of the major problems that need to be solved for visual programming to really work. Many people look at the problems and just throw up their hands, but I personally think they can be solved. I look at things like people building giant systems in Minecraft, something definitely not even designed for laying out logic or processing, and the way they talk about the physicality and geometry of the processing they lay out suggests to me that some breakthroughs might be right around the corner. Years ago when Supreme Commander was released (the game), they introduced an interface where you held down Ctrl and scrolled the mouse wheel in order to 'zoom' from high level overviews all the way down to single unit detail. It felt so intuitive and natural after using it for a bit that I found myself accidentally trying to do it in a file manager application after exiting the game. I think that would work for visual programming as well. And who knows the possibilities that VR brings along....
pierrec 8y ago
This seems great for creating smaller graphs as long as they don't get too complex. I'm kind of obsessed with node-based environments for DSP and media stuff (Reaktor, vvvv, and many others). Whenever my graphs attain some level of complexity, some features become very important such as selecting/operating on multiple nodes and most of all grouping. Rete.js doesn't seem to be quite there yet (no multi select? and Modules only hint at proper grouping), but it's definitely on the way.
- Immortalin 8y ago
  Try this: https://github.com/jagenjo/litegraph.js https://github.com/jagenjo/litegraph.js
kumarvvr 8y ago
Anyone know a similar thing in Python? Need it to integrate it into a project.

134 points by Immortalin 8y ago

gedy 8y ago

Really nice, haven't figured out what it's capable of, but reminds of a similar interface I wrote from scratch about 15 year ago in Java Swing. Blows me away how powerful browsers and JS have become. Really amazing that you can do this with so few lines of code, compared the (tens of?) thousands lines I wrote to do similar.

madrox 8y ago

Any time I see something like this I'm reminded of Yahoo Pipes and how ahead of its time it was. Is there anything like it that's designed for enterprise ETL these? It occurs to me it's a much better way of thinking about data flow problems than code.

oldmanhorton 8y ago
I think https://flow.microsoft.com/en-us/ https://flow.microsoft.com/en-us/ intends to be an enterprise version of this idea, but I've never used it.
- ptrott2017 8y ago
  for enterprise ETL - there is also kettle which has been around for a long time- which has a desktop and web authoring client. https://github.com/pentaho/pentaho-kettle https://github.com/pentaho/pentaho-kettle
agumonkey 8y ago
Node based / dataflow UI are also the bread and butter of CGI and compositing industry. It's hyper efficient both for machine because it's often freely memoized/cached and user (think of it as a simplified haskell at your fingertips) because the stateless and unlimited-undo aspects. I'm still pretty sure it should be the default UI for half computer introduction. Text for the rest, with the right paradigm of course.
- codetrotter 8y ago
  Node based user interfaces can be great indeed. Unreal Engine 4 has a visual scripting system called Blueprints that is very impressive. https://docs.unrealengine.com/en-US/Engine/Blueprints https://docs.unrealengine.com/en-US/Engine/Blueprints Another example of a node based user interface can be found in the open source 3d modeling and animation software Blender. https://docs.blender.org/manual/en/dev/render/blender_render/materials/nodes/introduction.html https://docs.blender.org/manual/en/dev/render/blender_render... The open source virtual modular synthesizer VCV Rack is node based as well, though it is so “by nature” in the sense that hardware modular synthesizer work in fashion of connecting inputs and outputs with patch cables. https://www.vcvrack.com/ https://www.vcvrack.com/ When done right, as all of the three I mentioned above in my opinion do, node based user interfaces are great. Not all node based user interfaces are equally good. I used LabView a bit at a course at the university and did not enjoy their system. The LabView node based user interface felt very clunky the whole time, even for doing simple things. I’ve been wondering occasionally whether certain kinds of data processing that I am doing with Unix tools could benefit from a graphical system. It’s not that I am not comfortable with the command line, but even as a rather die-hard user of Linux and FreeBSD I must admit that the command line has several shortcomings. Of course, designing a graphical successor to Unix pipelines is a major undertaking that will require through thought. But I think that keeping that idea in the back of ones mind while implementing some smaller node based systems could eventually lead to realization of most if not all of the considerations that must be kept in mind for the design of a graphical successor to Unix pipelines.
bpicolo 8y ago
Domo[0]. Their visual ETL is pretty good. https://www.domo.com/ https://www.domo.com/
monstrado 8y ago
Two come to mind. Apache NiFi: http://nifi.apache.org/ http://nifi.apache.org/ StreamSets: https://streamsets.com/opensource/ https://streamsets.com/opensource/ Side note, I'm an Apache NiFi committer and have had great success using it for enterprise ETL in a previous role.
- tmikaeld 8y ago
  Seems likestreamsets don't include a visual designer in the open source core? And no price, which seems standard with ETLs https://streamsets.com/subscriptions https://streamsets.com/subscriptions
pjmlp 8y ago
Yes, Bizztalk would be one of them.
statictype 8y ago
We have one that's a little more complex than Yahoo Pipes. Small Demo: https://s3.amazonaws.com/ecyber.public/lucy-use-cases/1/video.mp4 https://s3.amazonaws.com/ecyber.public/lucy-use-cases/1/vide... http://lucy.iviva.com http://lucy.iviva.com

feiss 8y ago

This is nice, great job! I started one framework very similar a couple of years ago, but never finished. I'm going to take a good look at this :)

c-rack 8y ago

The name reminds me on the Rete algorithm, which is pretty cool, but unrelated: https://en.wikipedia.org/wiki/Rete_algorithm https://en.wikipedia.org/wiki/Rete_algorithm

hodgesrm 8y ago
I found the name confusing for exactly this reason. Rete is a big deal in rule-based systems.
- kjhughes 8y ago
  Agreed. It's such a unique name for a classic algorithm and is so thoroughly established in symbolic pattern matching for production systems that I'd seriously recommend against adopting it for any other purpose in CS.

Kranar 8y ago

Is there a way to represent recursive nodes or recursion in general?

Ni55aN 8y ago
It can be represented visually, but the engine does not support processing recursive nodes

outofpaper 8y ago

Node Red is an other node based visual programming tool that is inspired by Yahoo pipes.

zoggenhoff 8y ago

Looks good. Time to brush up on my websockets and glue this into something.

freen 8y ago

The recurring problem I've found with node-based visual programming is that it's very challenging to work with multiple layers of abstraction and that refactoring can be incredibly challenging.

otakucode 8y ago
Those are definitely two of the major problems that need to be solved for visual programming to really work. Many people look at the problems and just throw up their hands, but I personally think they can be solved. I look at things like people building giant systems in Minecraft, something definitely not even designed for laying out logic or processing, and the way they talk about the physicality and geometry of the processing they lay out suggests to me that some breakthroughs might be right around the corner. Years ago when Supreme Commander was released (the game), they introduced an interface where you held down Ctrl and scrolled the mouse wheel in order to 'zoom' from high level overviews all the way down to single unit detail. It felt so intuitive and natural after using it for a bit that I found myself accidentally trying to do it in a file manager application after exiting the game. I think that would work for visual programming as well. And who knows the possibilities that VR brings along....

pierrec 8y ago

This seems great for creating smaller graphs as long as they don't get too complex. I'm kind of obsessed with node-based environments for DSP and media stuff (Reaktor, vvvv, and many others). Whenever my graphs attain some level of complexity, some features become very important such as selecting/operating on multiple nodes and most of all grouping. Rete.js doesn't seem to be quite there yet (no multi select? and Modules only hint at proper grouping), but it's definitely on the way.

Immortalin 8y ago
Try this: https://github.com/jagenjo/litegraph.js https://github.com/jagenjo/litegraph.js

kumarvvr 8y ago

Anyone know a similar thing in Python? Need it to integrate it into a project.