4 ms·
> dlt is a minimalistic Python library, requires no backend and works whenever Python works. You can finetune it to ... run with ... Dagster. Relating to dagst
by Terretta 3y ago
> dlt is a minimalistic Python library, requires no backend and works whenever Python works. You can finetune it to ... run with ... Dagster.
Relating to dagster in particular, this is in your docs:
dlt incorporates the concept of implicit extraction DAGs to handle the dependencies between data sources and their transformations automatically. A DAG represents a directed graph without cycles, where each node represents a data source or transformation step.
When using dlt, the tool automatically generates an extraction DAG based on the dependencies identified between the data sources and their transformations. This extraction DAG determines the optimal order for extracting the resources to ensure data consistency and integrity.
How do you think about tying and running this and dagster together?
- adrianbr 3y agoThere are multiple ways to run together - we will show a few in a demo coming out soon. We also consider a tighter integration like with Airflow described here as a possible next step https://dlthub.com/docs/walkthroughs/deploy-a-pipeline/deploy-with-airflow-composer https://dlthub.com/docs/walkthroughs/deploy-a-pipeline/deplo... We will investigate the interest incrementally as to not build any plugins that don't end up used.