4 ms·
Our hope with metaflow is to make the transition to production schedulers like Airflow (and perhaps similar technologies) seamless once you write the DAG via th
by seeravikiran 7y ago
Our hope with metaflow is to make the transition to production schedulers like Airflow (and perhaps similar technologies) seamless once you write the DAG via the FlowSpec.
The user doesn’t have to care about the conversion to YAML etc.
So I would say metaflow works in tandem with existing schedulers.
- sandGorgon 7y agoThis is very interesting as a goal. Are you saying metaflow is a "Jupyter notebook for airflow developers" kind of a thing ?
- seeravikiran 7y agoI wouldn't exactly say that. Jupyter notebooks don't have an easy way to represent an arbitrary DAG. The flow is more linear and narrative like. That said, we do expect metaflow (with client API) to play very well with notebooks to support a narrative from a business use-case pov; which might be the end-goal of most ML workloads (hopefully). I would like to think of metaflow, as your workflow construct - hopefully making your life simpler with ML workloads when involving interactions with existing pieces of infrastructure (infra pieces - storage, compute, notebooks or other UI, http service hosting etc.; concepts - collaboration, versioning, archiving, dependency management)
- sandGorgon 7y agosorry- i meant notebook figuratively. so metaflow is the local dev version of workflow construct. and then export that to airflow/etc compatible format ? what workflow engine do you guys use and primarily support in metaflow ?
- vtuulos 7y agothe workflow DAG is a core concept of Metaflow but Metaflow is not just that. Other core parts are managing and inspecting state, cloud integration, and organizing results. At Netflix, we use an internal workflow engine called Meson https://www.youtube.com/watch?v=0R58_tx7azY https://www.youtube.com/watch?v=0R58_tx7azY