7 ms·

Any time I see something like this I'm reminded of Yahoo Pipes and how ahead of its time it was. Is there anything like it that's designed for enterprise ETL th

by madrox 8y ago

Any time I see something like this I'm reminded of Yahoo Pipes and how ahead of its time it was. Is there anything like it that's designed for enterprise ETL these? It occurs to me it's a much better way of thinking about data flow problems than code.

oldmanhorton 8y ago
I think https://flow.microsoft.com/en-us/ https://flow.microsoft.com/en-us/ intends to be an enterprise version of this idea, but I've never used it.
- ptrott2017 8y ago
  for enterprise ETL - there is also kettle which has been around for a long time- which has a desktop and web authoring client. https://github.com/pentaho/pentaho-kettle https://github.com/pentaho/pentaho-kettle
agumonkey 8y ago
Node based / dataflow UI are also the bread and butter of CGI and compositing industry. It's hyper efficient both for machine because it's often freely memoized/cached and user (think of it as a simplified haskell at your fingertips) because the stateless and unlimited-undo aspects. I'm still pretty sure it should be the default UI for half computer introduction. Text for the rest, with the right paradigm of course.
- codetrotter 8y ago
  Node based user interfaces can be great indeed. Unreal Engine 4 has a visual scripting system called Blueprints that is very impressive. https://docs.unrealengine.com/en-US/Engine/Blueprints https://docs.unrealengine.com/en-US/Engine/Blueprints Another example of a node based user interface can be found in the open source 3d modeling and animation software Blender. https://docs.blender.org/manual/en/dev/render/blender_render/materials/nodes/introduction.html https://docs.blender.org/manual/en/dev/render/blender_render... The open source virtual modular synthesizer VCV Rack is node based as well, though it is so “by nature” in the sense that hardware modular synthesizer work in fashion of connecting inputs and outputs with patch cables. https://www.vcvrack.com/ https://www.vcvrack.com/ When done right, as all of the three I mentioned above in my opinion do, node based user interfaces are great. Not all node based user interfaces are equally good. I used LabView a bit at a course at the university and did not enjoy their system. The LabView node based user interface felt very clunky the whole time, even for doing simple things. I’ve been wondering occasionally whether certain kinds of data processing that I am doing with Unix tools could benefit from a graphical system. It’s not that I am not comfortable with the command line, but even as a rather die-hard user of Linux and FreeBSD I must admit that the command line has several shortcomings. Of course, designing a graphical successor to Unix pipelines is a major undertaking that will require through thought. But I think that keeping that idea in the back of ones mind while implementing some smaller node based systems could eventually lead to realization of most if not all of the considerations that must be kept in mind for the design of a graphical successor to Unix pipelines.
bpicolo 8y ago
Domo[0]. Their visual ETL is pretty good. https://www.domo.com/ https://www.domo.com/
monstrado 8y ago
Two come to mind. Apache NiFi: http://nifi.apache.org/ http://nifi.apache.org/ StreamSets: https://streamsets.com/opensource/ https://streamsets.com/opensource/ Side note, I'm an Apache NiFi committer and have had great success using it for enterprise ETL in a previous role.
- tmikaeld 8y ago
  Seems likestreamsets don't include a visual designer in the open source core? And no price, which seems standard with ETLs https://streamsets.com/subscriptions https://streamsets.com/subscriptions
pjmlp 8y ago
Yes, Bizztalk would be one of them.
statictype 8y ago
We have one that's a little more complex than Yahoo Pipes. Small Demo: https://s3.amazonaws.com/ecyber.public/lucy-use-cases/1/video.mp4 https://s3.amazonaws.com/ecyber.public/lucy-use-cases/1/vide... http://lucy.iviva.com http://lucy.iviva.com

by madrox 8y ago

oldmanhorton 8y ago

I think https://flow.microsoft.com/en-us/ https://flow.microsoft.com/en-us/ intends to be an enterprise version of this idea, but I've never used it.

ptrott2017 8y ago
for enterprise ETL - there is also kettle which has been around for a long time- which has a desktop and web authoring client. https://github.com/pentaho/pentaho-kettle https://github.com/pentaho/pentaho-kettle

agumonkey 8y ago

Node based / dataflow UI are also the bread and butter of CGI and compositing industry. It's hyper efficient both for machine because it's often freely memoized/cached and user (think of it as a simplified haskell at your fingertips) because the stateless and unlimited-undo aspects. I'm still pretty sure it should be the default UI for half computer introduction. Text for the rest, with the right paradigm of course.

codetrotter 8y ago
Node based user interfaces can be great indeed. Unreal Engine 4 has a visual scripting system called Blueprints that is very impressive. https://docs.unrealengine.com/en-US/Engine/Blueprints https://docs.unrealengine.com/en-US/Engine/Blueprints Another example of a node based user interface can be found in the open source 3d modeling and animation software Blender. https://docs.blender.org/manual/en/dev/render/blender_render/materials/nodes/introduction.html https://docs.blender.org/manual/en/dev/render/blender_render... The open source virtual modular synthesizer VCV Rack is node based as well, though it is so “by nature” in the sense that hardware modular synthesizer work in fashion of connecting inputs and outputs with patch cables. https://www.vcvrack.com/ https://www.vcvrack.com/ When done right, as all of the three I mentioned above in my opinion do, node based user interfaces are great. Not all node based user interfaces are equally good. I used LabView a bit at a course at the university and did not enjoy their system. The LabView node based user interface felt very clunky the whole time, even for doing simple things. I’ve been wondering occasionally whether certain kinds of data processing that I am doing with Unix tools could benefit from a graphical system. It’s not that I am not comfortable with the command line, but even as a rather die-hard user of Linux and FreeBSD I must admit that the command line has several shortcomings. Of course, designing a graphical successor to Unix pipelines is a major undertaking that will require through thought. But I think that keeping that idea in the back of ones mind while implementing some smaller node based systems could eventually lead to realization of most if not all of the considerations that must be kept in mind for the design of a graphical successor to Unix pipelines.

bpicolo 8y ago

Domo[0]. Their visual ETL is pretty good. https://www.domo.com/ https://www.domo.com/

monstrado 8y ago

Two come to mind. Apache NiFi: http://nifi.apache.org/ http://nifi.apache.org/ StreamSets: https://streamsets.com/opensource/ https://streamsets.com/opensource/ Side note, I'm an Apache NiFi committer and have had great success using it for enterprise ETL in a previous role.

tmikaeld 8y ago
Seems likestreamsets don't include a visual designer in the open source core? And no price, which seems standard with ETLs https://streamsets.com/subscriptions https://streamsets.com/subscriptions

pjmlp 8y ago

Yes, Bizztalk would be one of them.

statictype 8y ago

We have one that's a little more complex than Yahoo Pipes. Small Demo: https://s3.amazonaws.com/ecyber.public/lucy-use-cases/1/video.mp4 https://s3.amazonaws.com/ecyber.public/lucy-use-cases/1/vide... http://lucy.iviva.com http://lucy.iviva.com