3 ms·
This is why I though the shift from "orchestrate jobs" to "keep track of state of assets" that Dagster is trying to do is pretty important. But it sems it might
by glogla 4y ago
This is why I though the shift from "orchestrate jobs" to "keep track of state of assets" that Dagster is trying to do is pretty important. But it sems it might not be enough - it still keeps clunky (pythonic) interface and I don't know how well it does multi-tenancy.
- sethjr5rtfgh 4y agoCould you clarify what "keep track of state of assets" means?
- glogla 4y agohttps://www.dagster.io/blog/software-defined-assets https://www.dagster.io/blog/software-defined-assets Of course, nothing stops Airflow or other tools fron thinking this way as well.
- geoffjentry 4y agoIt's a mindset shift to a more declarative model. The idea has also popped up in other niche orchestrators. This is an oversimplification but IMO the easiest way of picturing it is instead thinking of defining your graph as a forward moving thing w/ the orchestrator telling things they can be run you shift to defining your graph nodes to know their dependencies and they let the orchestrator know when they're runnable.