4 ms·

Show HN: Airbyte, data integration platform with 300+ open-source connectors

29 points by Arimbr 4y ago

evantahler 4y ago
I don't think I've seen a YouTube badge on a README before. I like it!
bsnnkv 4y ago
I'm a Platorm person who has been tasked with setting this up for some data shovelling for a data team at work. Would not recommend. It's a pain to deploy/monitor and you'll have to abandon any hopes of reasonable/sane declarative configuration management. Everything eventually needs to be done through the UI, which is just buggy and painful.
- mritchie712 4y ago
  Are you sticking with it or using something else? Were you self hosting or using cloud? We're considering this vs. Fivetran.
  - bsnnkv 4y ago
    Hah. As if anyone listens to the concerns of Platform engineers. Product and data teams make their choices and we are left to pick up the pieces and glue them together into something passably functional. We are self hosting. Don't think the cloud hosted version is being considered anymore now that I got something up and running.
  - rorymalcolm 4y ago
    Switched to Fivetran after experiences with Airbyte, operating Airbyte was extremely tedious, even primetime connectors had bugs that I personally had to fix, we keep Airbyte around for the "long tail" of SaaS tools that Fivetran doesn't have a connector for but managing an Airbyte deployment in a stable manner for the long term was not easy.
- Arimbr 4y ago
  Know that there is also a CLI to manage configurations defined in YAML files. And a few options to deploy Airbyte in "one click". It's all in the README, sorry to hear you didn't find your way around our docs... There are a number of growing features and deployment options now.
  - bsnnkv 4y ago
    We are using the CLI, but it leaves a lot to be desired. Honestly, if you want to get serious about configuration management you need to put out a Terraform provider ASAP. The disaster recovery story in particular is very poor. To use this in production with confidence, we need to be able to spin up a brand new instance, and have a single command ("terraform apply" or equivalent), to apply the exact same configuration state (sources, destinations, connections) that was on a previously running instance.
    - Arimbr 4y ago
      Thanks for the feedback! Recently a community member created a Terraform provider: https://github.com/eabrouwer3/terraform-provider-airbyte https://github.com/eabrouwer3/terraform-provider-airbyte
    - bebop 4y ago
      Thanks for the feedback and we (Airbyte) are actively working on IaC solutions. We are publishing a Public API in 2023 [0]. We are working on an official Terraform provider built on that API, as well as, language specific SDKs. We hope to have these tools ready in 2023 as well. [0] - https://api.airbyte.com https://api.airbyte.com (There is a roadmap there).
    - rorymalcolm 4y ago
      Totally agree - in fact from the issues pages it seemed like there had been a conscious decision from Airbyte not to support a Terraform provider and instead build out Octavia, which as you say leaves a lot to be desired. Deploying Airbyte is still far too hard, I'd be happy to pay for an 'on prem' version like Gitlab where it's a bit more managed. I also think the failure modes for Airbyte on prem are too hard to debug in comparison to other data 'MDS' tools.
- ensemblehq 4y ago
  Thanks - just exploring Airbyte for a client so will be running through the motions myself. For another client, we just built our own data architecture to fix some of these issues and more importantly cut costs significantly. Were you able to find something else self-hostable? Ideally, we really want to stay away from cloud.
alexmarquardt 4y ago
Nice
shmoogy 4y ago
I setup airbyte cloud when I was tight on resources and needed integrations done fast, had no bandwidth to spend a day or two working on the ETL flows. Overall it's been good once it was configured. Ran into way too many issues for a paid platform though that gave me concern few people are actually using it in production. I did spin up a local instance with docker compose and you get some integrations not available in the cloud hosted version. I'm hesitant to give it a recommendation - their support is responsive and tries their best - eventually solving issues - but if something breaks and you lose visibility into marketing spend or something you NEED... it's not been 100% stable for me. E: to be fair to them - APIs change and break sometimes - I get it, I manage like 40 APIs in our airflow - but taking 3-5 days to fix something you need for production is risky.

29 points by Arimbr 4y ago

I don't think I've seen a YouTube badge on a README before. I like it!

I'm a Platorm person who has been tasked with setting this up for some data shovelling for a data team at work. Would not recommend. It's a pain to deploy/monitor and you'll have to abandon any hopes of reasonable/sane declarative configuration management. Everything eventually needs to be done through the UI, which is just buggy and painful.

mritchie712 4y ago
Are you sticking with it or using something else? Were you self hosting or using cloud? We're considering this vs. Fivetran.
- bsnnkv 4y ago
  Hah. As if anyone listens to the concerns of Platform engineers. Product and data teams make their choices and we are left to pick up the pieces and glue them together into something passably functional. We are self hosting. Don't think the cloud hosted version is being considered anymore now that I got something up and running.
- rorymalcolm 4y ago
  Switched to Fivetran after experiences with Airbyte, operating Airbyte was extremely tedious, even primetime connectors had bugs that I personally had to fix, we keep Airbyte around for the "long tail" of SaaS tools that Fivetran doesn't have a connector for but managing an Airbyte deployment in a stable manner for the long term was not easy.
Arimbr 4y ago
Know that there is also a CLI to manage configurations defined in YAML files. And a few options to deploy Airbyte in "one click". It's all in the README, sorry to hear you didn't find your way around our docs... There are a number of growing features and deployment options now.
- bsnnkv 4y ago
  We are using the CLI, but it leaves a lot to be desired. Honestly, if you want to get serious about configuration management you need to put out a Terraform provider ASAP. The disaster recovery story in particular is very poor. To use this in production with confidence, we need to be able to spin up a brand new instance, and have a single command ("terraform apply" or equivalent), to apply the exact same configuration state (sources, destinations, connections) that was on a previously running instance.
  - Arimbr 4y ago
    Thanks for the feedback! Recently a community member created a Terraform provider: https://github.com/eabrouwer3/terraform-provider-airbyte https://github.com/eabrouwer3/terraform-provider-airbyte
  - bebop 4y ago
    Thanks for the feedback and we (Airbyte) are actively working on IaC solutions. We are publishing a Public API in 2023 [0]. We are working on an official Terraform provider built on that API, as well as, language specific SDKs. We hope to have these tools ready in 2023 as well. [0] - https://api.airbyte.com https://api.airbyte.com (There is a roadmap there).
  - rorymalcolm 4y ago
    Totally agree - in fact from the issues pages it seemed like there had been a conscious decision from Airbyte not to support a Terraform provider and instead build out Octavia, which as you say leaves a lot to be desired. Deploying Airbyte is still far too hard, I'd be happy to pay for an 'on prem' version like Gitlab where it's a bit more managed. I also think the failure modes for Airbyte on prem are too hard to debug in comparison to other data 'MDS' tools.
ensemblehq 4y ago
Thanks - just exploring Airbyte for a client so will be running through the motions myself. For another client, we just built our own data architecture to fix some of these issues and more importantly cut costs significantly. Were you able to find something else self-hostable? Ideally, we really want to stay away from cloud.

alexmarquardt 4y ago

Nice

shmoogy 4y ago

I setup airbyte cloud when I was tight on resources and needed integrations done fast, had no bandwidth to spend a day or two working on the ETL flows. Overall it's been good once it was configured. Ran into way too many issues for a paid platform though that gave me concern few people are actually using it in production. I did spin up a local instance with docker compose and you get some integrations not available in the cloud hosted version. I'm hesitant to give it a recommendation - their support is responsive and tries their best - eventually solving issues - but if something breaks and you lose visibility into marketing spend or something you NEED... it's not been 100% stable for me. E: to be fair to them - APIs change and break sometimes - I get it, I manage like 40 APIs in our airflow - but taking 3-5 days to fix something you need for production is risky.