Yes I think as a community we have largely got our levels of abstraction incorrect: - Code for pipelines without frameworks leads to huge repetition of logic -

Yes I think as a community we have largely got our levels of abstraction incorrect: - Code for pipelines without frameworks leads to huge repetition of logic - or worse people reimplementing the same 'logic' differently. Also you end up with a massive upgrade problem when new versions of underling execution engines change. - Databricks provides a low-level API which leads to a lot of duplication of common code across notebooks (and reusability is difficult). - The GUI based tools are often too high level so have very high reusability but are difficult to customise - and hard to source control. We have tried to build an abstraction somewhere in between which gives you the reusability of the GUI tools, plays nicely with source control and has the power to add custom logic via the plugin interface: https://arc.tripl.ai/plugins/ https://arc.tripl.ai/plugins/ if required. (formatting)