3 ms·
Let me give you some bullets on DataFactory (DF) because it is a question I get a lot. - DF is quite hard to operationalize, the logging is not so good, Python
by kfk 3y ago
Let me give you some bullets on DataFactory (DF) because it is a question I get a lot.
- DF is quite hard to operationalize, the logging is not so good, Python stack traces are easier to debug and logging can get as detailed as needed
- DF lacks connectors for data ingestion, this is easy in Python as on average a custom connector takes a week or two to develop
- DF is not data pipelines as code and it is becoming really hard to manage governance and change management on UI based ETL tools
- It is hard to enforce best practices on DF. We are finding it is easy to enforce standard ways of writing and managing SQL models and metadata with a combo of dbt and a dbt-ready data catalog