3 ms·

I am considering/designing/PoCing implementing a low-code (visual programming like Unreal) system for a specialized ETL (algorithmic) system. Both E and L would

I am considering/designing/PoCing implementing a low-code (visual programming like Unreal) system for a specialized ETL (algorithmic) system. Both E and L would be nodes and the target audience mostly uses math in the T. Great advantage seems to be an ability to schedule jobs based on algorithm needs, as everything is known in the graph. I also think we can give some guarantees like infinite loop prevention by analysing the graph. An alternative I see is a textual DSL which may be even more work to implement and user discovery becomes harder. I’d love to talk and discuss about this with others; it is hard for me to determine whether this is the way to go with so many opinions in either camp.

techdragon 3y ago
ETL is one of the places that NoCode LowCode solutions and Visual Programming systems have been successful despite the amount of negative information out there. They’re a sort of silent success story in the ETL works since 1: it’s not glamorous work so there’s not a lot to brag about 2: it’s often solved by just paying for some proprietary vendor solution since they will have already done the hard work getting the E and L parts done and plenty of companies have built compelling sales pitches about how simple the T will be with their drag and drop design tools and 3: once the ETL pipeline is done, a few revisions is usually all they get in most workplaces, and then it will run in the background silently until someone questions the cost of running it or asks where the data in some system is coming from or why they get some emailed reports. The ETL niche is a good fit for the kind of approach you’re describing, so I don’t see any reason not to build at least a prototype to see how the end users like the design … beyond the obvious complexity of getting a visual node based editor and data format up and going (and there’s some open source out there that makes that part not as much work as it would have been 5-10 years ago)

techdragon 3y ago

ETL is one of the places that NoCode LowCode solutions and Visual Programming systems have been successful despite the amount of negative information out there. They’re a sort of silent success story in the ETL works since 1: it’s not glamorous work so there’s not a lot to brag about 2: it’s often solved by just paying for some proprietary vendor solution since they will have already done the hard work getting the E and L parts done and plenty of companies have built compelling sales pitches about how simple the T will be with their drag and drop design tools and 3: once the ETL pipeline is done, a few revisions is usually all they get in most workplaces, and then it will run in the background silently until someone questions the cost of running it or asks where the data in some system is coming from or why they get some emailed reports. The ETL niche is a good fit for the kind of approach you’re describing, so I don’t see any reason not to build at least a prototype to see how the end users like the design … beyond the obvious complexity of getting a visual node based editor and data format up and going (and there’s some open source out there that makes that part not as much work as it would have been 5-10 years ago)