4 ms·

UI based flows for integrations are as bad as UI based flows for ETL. For some reason, we managed to move past UI to code for ETL, but now we are taking the exa

by kfk 4y ago

UI based flows for integrations are as bad as UI based flows for ETL. For some reason, we managed to move past UI to code for ETL, but now we are taking the exact same path for integrations. Integrations are complex, you might have complex requirements, polling and pushing needs depending on the applications, fast lookups needs, legacy IT solutions that hold critical data. I don't think we are ready to go full UI in this space, I think good orchestration and good microservices are still a better choice.

supersync 4y ago
Check out Make, n8n, workflow86 Agree that coded integrations are still very relevant, but the addressable surface for UI is pretty large now.
- ngz00 4y ago
  Is there any open source, embeddable alternatives or frameworks? I've got an ETL DSL that is fully serializable and I've been looking to create a UI tool for it.
  - iamacyborg 4y ago
    Maybe Airbyte if I understand what you're asking correctly.
    - georgesequeira 4y ago
      Ah right. Yeah, reverse-engineering Airbyte's DSL and just "borrowing" their UI generation may be the move.
  - georgesequeira 4y ago
    I don't know of a solution for ETL DSL -> UI but https://github.com/lowdefy/lowdefy https://github.com/lowdefy/lowdefy goes from a DSL to internal web apps. Perhaps some inspiration could be taken from there. Good luck!
    - ngz00 4y ago
      Thats pretty fascinating stuff, thanks for the recommendation.
tmp_anon_22 4y ago
> UI based flows for integrations are as bad Right but in the same breath we should mention all code is terrible.
andybak 4y ago
How about catering to non-programmers?
- kfk 4y ago
  There is a space for that for sure, but when things get serious, the programming bit is 10% of the capability, domain knowledge, data expertise and SQL become more relevant. Those skills are not fulfilled with UI tools. Sometimes in fact UI tools make the skill gap bigger because new hires might know SQL, but for sure don't know your expensive proprietary tool of choice.
PaulHoule 4y ago
Yeah, this is almost the sentiment behind the snarky comment I made that got voted down. The difference is I am not so pessimistic about those UI-based ETL systems because I did a deep dive on the architecture of those and even worked for a startup that built one. I've seen how the industry's fixation on raw performance has kept ETL systems using a relational model that is hell to program for complex jobs -- I pitched a next generation system to a very acquisitive leader in this space and got shot down because my system wasn't friendly to storage locality and SIMD instructions. (That startup had the right idea of passing 'JSON' documents between the lines and the boxes but never settled exactly what the algebra was for those things, how to close event pipelines in the end properly so you always get the right results, ...) Those "integration builders" though strike me a way of replacing a 10 line Python script with incredible tedium and "you can't get from here to there" experiences. That leads me to say things like "When I hear the word 'integration' I reach for my gun."
tomatowurst 4y ago
Depends where those UIs are on. If you have some sort of agreement set in place where they won't update the interface, then this is very much possible. If it is relying on an 3rd party platform that you can't control and not know when UI changes are coming, then it can be bad for criticality. However, I fail to know how urgent criticality is in ETL space, especially when UI changes can be as simple as "theres a new button" or "margin on form changed". The key process features and UI related to it would rarely change.
te_chris 4y ago
UI is bad for ETL...huh? I hope you're not talking about Stitch/Fivetran, because they're a godsend in my experience.
intrasight 4y ago
Zapier could easily add a DSL that mirrors their UI. It's not the UI that makes it powerful, it's the integrations and workflow. My approach to integration for ETL was a DSL.

by kfk 4y ago

supersync 4y ago

Check out Make, n8n, workflow86 Agree that coded integrations are still very relevant, but the addressable surface for UI is pretty large now.

ngz00 4y ago
Is there any open source, embeddable alternatives or frameworks? I've got an ETL DSL that is fully serializable and I've been looking to create a UI tool for it.
- iamacyborg 4y ago
  Maybe Airbyte if I understand what you're asking correctly.
  - georgesequeira 4y ago
    Ah right. Yeah, reverse-engineering Airbyte's DSL and just "borrowing" their UI generation may be the move.
- georgesequeira 4y ago
  I don't know of a solution for ETL DSL -> UI but https://github.com/lowdefy/lowdefy https://github.com/lowdefy/lowdefy goes from a DSL to internal web apps. Perhaps some inspiration could be taken from there. Good luck!
  - ngz00 4y ago
    Thats pretty fascinating stuff, thanks for the recommendation.

tmp_anon_22 4y ago

> UI based flows for integrations are as bad Right but in the same breath we should mention all code is terrible.

andybak 4y ago

How about catering to non-programmers?

kfk 4y ago
There is a space for that for sure, but when things get serious, the programming bit is 10% of the capability, domain knowledge, data expertise and SQL become more relevant. Those skills are not fulfilled with UI tools. Sometimes in fact UI tools make the skill gap bigger because new hires might know SQL, but for sure don't know your expensive proprietary tool of choice.

PaulHoule 4y ago

Yeah, this is almost the sentiment behind the snarky comment I made that got voted down. The difference is I am not so pessimistic about those UI-based ETL systems because I did a deep dive on the architecture of those and even worked for a startup that built one. I've seen how the industry's fixation on raw performance has kept ETL systems using a relational model that is hell to program for complex jobs -- I pitched a next generation system to a very acquisitive leader in this space and got shot down because my system wasn't friendly to storage locality and SIMD instructions. (That startup had the right idea of passing 'JSON' documents between the lines and the boxes but never settled exactly what the algebra was for those things, how to close event pipelines in the end properly so you always get the right results, ...) Those "integration builders" though strike me a way of replacing a 10 line Python script with incredible tedium and "you can't get from here to there" experiences. That leads me to say things like "When I hear the word 'integration' I reach for my gun."

tomatowurst 4y ago

Depends where those UIs are on. If you have some sort of agreement set in place where they won't update the interface, then this is very much possible. If it is relying on an 3rd party platform that you can't control and not know when UI changes are coming, then it can be bad for criticality. However, I fail to know how urgent criticality is in ETL space, especially when UI changes can be as simple as "theres a new button" or "margin on form changed". The key process features and UI related to it would rarely change.

te_chris 4y ago

UI is bad for ETL...huh? I hope you're not talking about Stitch/Fivetran, because they're a godsend in my experience.

intrasight 4y ago

Zapier could easily add a DSL that mirrors their UI. It's not the UI that makes it powerful, it's the integrations and workflow. My approach to integration for ETL was a DSL.