3 ms·
It's intended for syncing data sources to a data warehouse for analysis. Even if you're only doing one data source, if you want analysis to be easy later you'll
by markmacardle 6y ago
It's intended for syncing data sources to a data warehouse for analysis. Even if you're only doing one data source, if you want analysis to be easy later you'll likely want to sync all fields available from the api (as you probably don't know what fields are of interest before analysis).
If you imagine doing this for Stripe say, there's a huge amount of fields available in different objects (charges, invoices, subscriptions etc) and you need to add these as columns to your relation database of a data warehouse. Unnesting may also be needed. That's very tedious work and on top of it you need to run, monitor and maintain the extraction process.
Even a small company could easily have 10+ data sources containing dozens of tables that they want to sync to a warehouse so this quickly becomes unmanageable. Hence companies like Stitchdata, Fivetran and now Airbyte now selling it as a service.