4 ms·

Know that there is also a CLI to manage configurations defined in YAML files. And a few options to deploy Airbyte in "one click". It's all in the README, sorry

by Arimbr 4y ago

Know that there is also a CLI to manage configurations defined in YAML files. And a few options to deploy Airbyte in "one click". It's all in the README, sorry to hear you didn't find your way around our docs... There are a number of growing features and deployment options now.

bsnnkv 4y ago
We are using the CLI, but it leaves a lot to be desired. Honestly, if you want to get serious about configuration management you need to put out a Terraform provider ASAP. The disaster recovery story in particular is very poor. To use this in production with confidence, we need to be able to spin up a brand new instance, and have a single command ("terraform apply" or equivalent), to apply the exact same configuration state (sources, destinations, connections) that was on a previously running instance.
- Arimbr 4y ago
  Thanks for the feedback! Recently a community member created a Terraform provider: https://github.com/eabrouwer3/terraform-provider-airbyte https://github.com/eabrouwer3/terraform-provider-airbyte
- bebop 4y ago
  Thanks for the feedback and we (Airbyte) are actively working on IaC solutions. We are publishing a Public API in 2023 [0]. We are working on an official Terraform provider built on that API, as well as, language specific SDKs. We hope to have these tools ready in 2023 as well. [0] - https://api.airbyte.com https://api.airbyte.com (There is a roadmap there).
- rorymalcolm 4y ago
  Totally agree - in fact from the issues pages it seemed like there had been a conscious decision from Airbyte not to support a Terraform provider and instead build out Octavia, which as you say leaves a lot to be desired. Deploying Airbyte is still far too hard, I'd be happy to pay for an 'on prem' version like Gitlab where it's a bit more managed. I also think the failure modes for Airbyte on prem are too hard to debug in comparison to other data 'MDS' tools.

by Arimbr 4y ago

bsnnkv 4y ago

We are using the CLI, but it leaves a lot to be desired. Honestly, if you want to get serious about configuration management you need to put out a Terraform provider ASAP. The disaster recovery story in particular is very poor. To use this in production with confidence, we need to be able to spin up a brand new instance, and have a single command ("terraform apply" or equivalent), to apply the exact same configuration state (sources, destinations, connections) that was on a previously running instance.

Arimbr 4y ago
Thanks for the feedback! Recently a community member created a Terraform provider: https://github.com/eabrouwer3/terraform-provider-airbyte https://github.com/eabrouwer3/terraform-provider-airbyte
bebop 4y ago
Thanks for the feedback and we (Airbyte) are actively working on IaC solutions. We are publishing a Public API in 2023 [0]. We are working on an official Terraform provider built on that API, as well as, language specific SDKs. We hope to have these tools ready in 2023 as well. [0] - https://api.airbyte.com https://api.airbyte.com (There is a roadmap there).
rorymalcolm 4y ago
Totally agree - in fact from the issues pages it seemed like there had been a conscious decision from Airbyte not to support a Terraform provider and instead build out Octavia, which as you say leaves a lot to be desired. Deploying Airbyte is still far too hard, I'd be happy to pay for an 'on prem' version like Gitlab where it's a bit more managed. I also think the failure modes for Airbyte on prem are too hard to debug in comparison to other data 'MDS' tools.