3 ms·
Data pipelines are not workflows
- remram 4y agoThe actual title is "Declarative Scheduling for Data Assets" > please use the original title, unless it is misleading or linkbait; don't editorialize
- s_ryz 4y agoTrying to represent and schedule pipelines of data assets as workflows is painful, because workflows are an awkward metaphor for data pipelines. A single data source can power dozens of data products, each with different data freshness requirements. A single data product can consume data from dozens of different sources, each of which is updated at a different cadence. Trying to chop up this graph of data assets into discrete workflows that each get executed synchronously often feels like forcing a square peg into a round hole. Dagster 1.1 introduces a system for scheduling data pipelines that lets you escape writing workflows entirely. Instead, you directly specify how up-to-date you expect each asset to be, as well as how to determine whether source data has changed. Dagster then automatically schedules asset materializations to ensure that data arrives on time, while avoiding unnecessary computation.
- rnk 4y agoThis seems similar to the data mesh concepts. Can you contrast what you are describing here with data mesh?
- deleted 4y ago[deleted]
- hobofan 4y agoI really like the vision this post lays out! I'm building out a similar system at my job right now, and as the next level of sophistication of that is on the horizon in the next few months, I'm wondering how difficult it would be to adopt Dagster. Looking at the documentation Dagster looks incredibly Python focused. Is there any way to use it without having to write a single line of Python, and rather generating declaration files of a fixed schema via other languages and using the dagster server as a black box?
- s_ryz 4y agoAlas, in Dagster's current form, you need to write Python.
- voz_ 4y agoThis is an ad.