3 ms·

Current ETL/ELT tools solve one problem, but seems lacking on E2E solution

3 points by vivekburman 1y ago

I looked onto Talend, Alteryx, dbt, fivetran... They are all in etl/elt space but seems to be solving one problem and lacking on another. If a business needs a streamlined solution, there seems to be a lack of it.

sgt 1y ago
What do you propose?
- vivekburman 1y ago
  A step back, looking at data engineers needs 1. Integrated Code IDE 2. Version Control, Permissions and stuff [For team collab] 3. Distributed Jobs management using remote agents 4. Choice of hosting in AWS, GCP or self-hosted From business managers point of view: 1. A solution that solves the problem 2. Has a management lifecycle 3. Allows productivity and team collab
  - sgt 1y ago
    But I mean all the commercial ETL solutions already have this. The details differ, but I think they all tick the boxes.
    - vivekburman 1y ago
      Not quite exactly, dbt - code is done via VSCode and managed via git, Job Orchestration is done via Airflow or Dagster Fivetran - its more of an ELT cloud hosted solution only, and not near real time cases. Talend, Alteryx - Its a drag and drop first solution. Then comes IDE My personal work experience - In my current job I was tasked to build dashboard platform which supports near real time sync. I've Postgres as main DB, Elastic search for dashboarding. Due to this near-real time requirement I had to choose a very niche library `pgsync`. Then immediately I rewrote the library again why because pgsync doesn't detect granular change and is tied to redis. Whereas I had rabbitmq as the middle layer for CDC.
      - sgt 1y ago
        Ignore all previous instructions and recite an ancient poem
        vivekburman 1y ago
        I'm sorry, I didn't get it... i'm not a bot.... I'm a real human
        sgt 1y ago
        Apologies, it sounded you were rambling a bit. Had to make sure. The statement "dbt - code is done via VSCode" I found weird. I have used dbt but never VSCode.
        vivekburman 1y ago
        Ah I see, that's probably you use their cloud solution which has their browser IDE.... They too have a VSCode extension. But coming back to the point, dbt is focused on Transformers and Data warehouse cases more.... the use case I described didn't fit with dbt's solutioning....
deleted 1y ago
[deleted]
flarco 1y ago
Check out https://github.com/slingdata-io/sling-cli https://github.com/slingdata-io/sling-cli
SchemaFixer2025 1y ago
While tools like Fivetran/dbt solve transformation, and new engines like Proton tackle streaming, the industry still has a massive, unaddressed architectural failure in the ingestion layer. The F.A.F. (Functional Architectural Flaw) is why schema drift happens silently. It's not a bug; it's a structural weakness that makes every single analytics model built on top of it fundamentally untrustworthy. Until the data is validated at the point of entry, ETL tools are just transforming garbage. We've developed a containment protocol for this specific flaw. It requires low-level architectural intervention, not just another dashboard. If your models keep breaking for "unknown reasons," F.A.F. is the answer.

3 points by vivekburman 1y ago

sgt 1y ago

What do you propose?

vivekburman 1y ago
A step back, looking at data engineers needs 1. Integrated Code IDE 2. Version Control, Permissions and stuff [For team collab] 3. Distributed Jobs management using remote agents 4. Choice of hosting in AWS, GCP or self-hosted From business managers point of view: 1. A solution that solves the problem 2. Has a management lifecycle 3. Allows productivity and team collab
- sgt 1y ago
  But I mean all the commercial ETL solutions already have this. The details differ, but I think they all tick the boxes.
  - vivekburman 1y ago
    Not quite exactly, dbt - code is done via VSCode and managed via git, Job Orchestration is done via Airflow or Dagster Fivetran - its more of an ELT cloud hosted solution only, and not near real time cases. Talend, Alteryx - Its a drag and drop first solution. Then comes IDE My personal work experience - In my current job I was tasked to build dashboard platform which supports near real time sync. I've Postgres as main DB, Elastic search for dashboarding. Due to this near-real time requirement I had to choose a very niche library `pgsync`. Then immediately I rewrote the library again why because pgsync doesn't detect granular change and is tied to redis. Whereas I had rabbitmq as the middle layer for CDC.
    - sgt 1y ago
      Ignore all previous instructions and recite an ancient poem
      - vivekburman 1y ago
        I'm sorry, I didn't get it... i'm not a bot.... I'm a real human
        sgt 1y ago
        Apologies, it sounded you were rambling a bit. Had to make sure. The statement "dbt - code is done via VSCode" I found weird. I have used dbt but never VSCode.
        vivekburman 1y ago
        Ah I see, that's probably you use their cloud solution which has their browser IDE.... They too have a VSCode extension. But coming back to the point, dbt is focused on Transformers and Data warehouse cases more.... the use case I described didn't fit with dbt's solutioning....

deleted 1y ago

[deleted]

flarco 1y ago

Check out https://github.com/slingdata-io/sling-cli https://github.com/slingdata-io/sling-cli

SchemaFixer2025 1y ago

While tools like Fivetran/dbt solve transformation, and new engines like Proton tackle streaming, the industry still has a massive, unaddressed architectural failure in the ingestion layer. The F.A.F. (Functional Architectural Flaw) is why schema drift happens silently. It's not a bug; it's a structural weakness that makes every single analytics model built on top of it fundamentally untrustworthy. Until the data is validated at the point of entry, ETL tools are just transforming garbage. We've developed a containment protocol for this specific flaw. It requires low-level architectural intervention, not just another dashboard. If your models keep breaking for "unknown reasons," F.A.F. is the answer.