3 ms·

Yeah me too. I think visualizations are cool and can be useful in terms of monitoring the progress of analysis runs. But I can't see my self using a GUI for set

Yeah me too. I think visualizations are cool and can be useful in terms of monitoring the progress of analysis runs. But I can't see my self using a GUI for setting up pipelines when I can do it more explicitly through scripting or some kind of a command line based pipeline framework. However, maybe the use case here is for team collaboration where the pipelines are easier conveyed through visualizations.

twunde 6y ago
There's a large tradition of data analysis/BI tools using visual programming/ "no-code" tools, think Talend, etc. They're popular since data analysis typically contains a number of people without coding backgrounds. Even among data science teams, many don't have the engineering background of data engineers (and certainly there's a segment of data scientists who don't care as much about building the data pipelines and the cleaning that's necessary). Low code tools _can_ (but not necessarily will) effectively empower data teams.
yannickperrenet 6y ago
Thanks for your input. We are planning to also support an API and/or CLI (besides the GUI) to set up the pipelines through scripting. Depending on what we hear from the community we will definitely add this sooner rather than later. You make a great point for the use of monitoring the progress of a run. During pipeline runs the visual pipeline interface (also in view only mode during or after running pipeline runs as part of an experiment) can be helpful as a visual indication of where the pipeline failed, how long pipeline steps took to execute and whether certain pipeline steps are still running. Lastly, for team collaboration we think the visual interface is indeed helpful to explain (especially to the less technical people) the structure of the code, what exactly the pipeline sets out to solve and how the pipeline solves it. In addition, the possibility to add intermediary pipeline steps that serve as dashboards can be helpful for collaboration.

twunde 6y ago

There's a large tradition of data analysis/BI tools using visual programming/ "no-code" tools, think Talend, etc. They're popular since data analysis typically contains a number of people without coding backgrounds. Even among data science teams, many don't have the engineering background of data engineers (and certainly there's a segment of data scientists who don't care as much about building the data pipelines and the cleaning that's necessary). Low code tools _can_ (but not necessarily will) effectively empower data teams.

yannickperrenet 6y ago

Thanks for your input. We are planning to also support an API and/or CLI (besides the GUI) to set up the pipelines through scripting. Depending on what we hear from the community we will definitely add this sooner rather than later. You make a great point for the use of monitoring the progress of a run. During pipeline runs the visual pipeline interface (also in view only mode during or after running pipeline runs as part of an experiment) can be helpful as a visual indication of where the pipeline failed, how long pipeline steps took to execute and whether certain pipeline steps are still running. Lastly, for team collaboration we think the visual interface is indeed helpful to explain (especially to the less technical people) the structure of the code, what exactly the pipeline sets out to solve and how the pipeline solves it. In addition, the possibility to add intermediary pipeline steps that serve as dashboards can be helpful for collaboration.