3 ms·

Is this meant as a free version of say DataFactory/FiveTran? I’m always interested in how these products do the change tracking and incremental updates. FiveTra

Is this meant as a free version of say DataFactory/FiveTran? I’m always interested in how these products do the change tracking and incremental updates. FiveTran is seamless, DF, more manual configuration. I see they don’t even support incremental updates for MSSQL/PostgreSQL so I’m wondering who would be using this tool?

sdepablos 6y ago
Sounds kind of strange not having incremental replication, it would make no sense for an ETL tool. In fact the documentation for PostgreSQL [1] is somehow contradictory at it both states "Incremental sync (copying only the data that has changed) for this source is coming soon" but also "Feature: Incremental - Append Sync / Supported: Yes" [1] https://docs.airbyte.io/integrations/sources/postgres https://docs.airbyte.io/integrations/sources/postgres
- jeanlaf 6y ago
  (Airbyte engineer) Agreed. Thanks for notifying us about that. We fixed it. We started by supporting only full refresh at the very start, as it was easier to support more connectors, but have been ramping up incremental across all the sources for a month now. Incremental should be supported for all sources within a few weeks.
sherifnada 6y ago
Hi, Airybte engineer here! We do support incremental updates for MSSQL & Postgres -- I updated the docs to make this clear. More broadly as an ELT tool, Airbyte is focused on supporting the long tail of integrations. The number of data silos is increasing rapidly, and it's intractable for a single engineering team or company to integrate with them all. So as an OSS tool, Airbyte provides a core platform for replicating data (e.g: a UI, scheduling, logging, state management, etc..) and the most popular connectors out of the box, then gets out of the way of its users. If you need a connector you need that Airbyte doesn't already provide, you can implement a Docker image which conforms to the Airbyte protocol, then start using it in your Airbyte instance without ever needing to go through us (Although many users do end up contributing their connectors back to the core codebase).

sdepablos 6y ago

Sounds kind of strange not having incremental replication, it would make no sense for an ETL tool. In fact the documentation for PostgreSQL [1] is somehow contradictory at it both states "Incremental sync (copying only the data that has changed) for this source is coming soon" but also "Feature: Incremental - Append Sync / Supported: Yes" [1] https://docs.airbyte.io/integrations/sources/postgres https://docs.airbyte.io/integrations/sources/postgres

jeanlaf 6y ago
(Airbyte engineer) Agreed. Thanks for notifying us about that. We fixed it. We started by supporting only full refresh at the very start, as it was easier to support more connectors, but have been ramping up incremental across all the sources for a month now. Incremental should be supported for all sources within a few weeks.

sherifnada 6y ago

Hi, Airybte engineer here! We do support incremental updates for MSSQL & Postgres -- I updated the docs to make this clear. More broadly as an ELT tool, Airbyte is focused on supporting the long tail of integrations. The number of data silos is increasing rapidly, and it's intractable for a single engineering team or company to integrate with them all. So as an OSS tool, Airbyte provides a core platform for replicating data (e.g: a UI, scheduling, logging, state management, etc..) and the most popular connectors out of the box, then gets out of the way of its users. If you need a connector you need that Airbyte doesn't already provide, you can implement a Docker image which conforms to the Airbyte protocol, then start using it in your Airbyte instance without ever needing to go through us (Although many users do end up contributing their connectors back to the core codebase).