4 ms·
1 - Hide your ETL logic in REST APIs and use common microservice patterns for monitoring, logging, etc... 2 - Use PythonOperator to call your APIs. It's worki
by pid-1 5y ago
1 - Hide your ETL logic in REST APIs and use common microservice patterns for monitoring, logging, etc...
2 - Use PythonOperator to call your APIs.
It's working really well for us.
- kdazzle 5y agoI used to do that, but having to deal with timeouts in long running jobs was frustrating because i never knew if something was successful. Could bump up the timeouts, but theyre there for a reason.
- ibishvintilli 5y agoThis is a very bad idea. You cannot hide ETL in a Rest API, at most you can hide the T part and it's a lot more work. you need to build an extra service that reads the data from a database and does some padgination. It's way more complex than a one-liner to get data from a jdbc connection and way much slower.