3 ms·
It's a bit unclear on what the scope of this tool is. But there are a couple of new classes of tools for ETL/ELT or data engineering as it's called now. There
by forgetfulness 6y ago
It's a bit unclear on what the scope of this tool is.
But there are a couple of new classes of tools for ETL/ELT or data engineering as it's called now.
There's the "Data Integration Tools" like Fivetran, Stitch, and this. They are collections of connectors that they have coded to ease ingesting data from lots of different database products and stores to another. That's valuable, I wouldn't start writing my own script to pull changes from my RDBMS' WAL to the data warehouse because it's complicated, and if someone can pull those JSON files from whatever cloud storage for me it's good because it's too simple to waste time on.
Then there's DBT and Dataform, those are orchestration tools and development environments all in one. When you start splitting your scripts (SQL queries in this case) in stages either for ease of understanding or efficiency, you'll want to see the dependencies laid out and having them execute in order. They also provide (git) version control, so it makes it a breeze for engineers to manage the pipeline like they would with other software assets, and you can get people from other backgrounds contributing in a more engineering-like workflow, talking about analysts who devise business dashboards and such.
- bleonard 6y agoJust for fun and completeness, there's also taking action on this data outside of your warehouse by making your external tools smarter - in some cases, even writing _back_ to the things you EL'd into it. For example, writing product usage to Salesforce. These are the kinds of things we are focusing on at Grouparoo, also an open source project.
- xupybd 6y agoThanks that makes a lot of sense.