4 ms·
We currently are using Airflow for ELTs/ETLs to ingest from different Postgres databases to BigQuery/Google Cloud Storage. Airbyte looks sweet for the same task
by mrbungie 5y ago
We currently are using Airflow for ELTs/ETLs to ingest from different Postgres databases to BigQuery/Google Cloud Storage. Airbyte looks sweet for the same task and would free us from a big effort burden, but its Postgres source only supports SELECT * statements (i.e. you can't deselect columns).
That's kind of a dealbreaker for us, because for security reasons our Postgres users permissions are granularly configured with column-based security. I hope the Airbyte team solves this eventually because the software is looking great.
- lloydatkinson 5y agoThat's crazy, why would it have the approach of "select all columns" when you might never need all of them? Could you create a view for it to select from instead? Either way if someone told me a ETL supports only literally all the columns or nothing then I'd assume it was a MVP product to demonstrate the idea... not a production system.
- davinchia 5y agoWe are working on it. This should be out next quarter!
- lixtra 5y agoIt’s strange to advertise a framework that doesn’t support more than “select *”. A framework should make easy things easy and complicated things possible. Ideally there is a natural way to do stuff, so different people understand each other’s approach. The whole article sounds like you target total amateurs in the ETL domain. Every decent ETL engineer knows all of that what is revealed after “a few months”. That said, I always welcome new competition in the field. Tools still suck.
- banku_brougham 5y agoThats crazy. Lets see what they to do to fill out this area, somebody has surely filed an issue on the girhub repo be sure to upvote it.