3 ms·
Metaflow comes with a built in scheduler. If your company has an existing scheduler, e.g. for ETL, you can translate Metaflow DAGs to the production scheduler a
by vtuulos 7y ago
Metaflow comes with a built in scheduler. If your company has an existing scheduler, e.g. for ETL, you can translate Metaflow DAGs to the production scheduler automatically. This is what we do at Netflix.
We could provide similar support e.g. for Airflow, if there's interest.
For monitoring, we have relied on notebooks this far. Their flexibility and customizability is unbeatable. We might build some lightweight discovery / debugging UI later, if there's demand. We are a bit on the fence about it internally.
- thundergolfer 7y ago> Metaflow comes with a built in scheduler. This isn't a scheduler that will trigger 'flows' though is it? If I search "Scheduler" in your docs the top result is a roadmap item, and searching "Cron" turns up nothing. Is it true that right now that to run a DAG "every day at 3am UTC" requires an external service?
- pela 7y agoRight now, yes.
- thundergolfer 7y agoThanks for confirmation.
- thundergolfer 7y ago> For monitoring, we have relied on notebooks this far. This is pretty interesting approach. Notebooks are a natural environment of Data Scientist and for monitoring they'd provide really good discoverability, flexibility, and interactivity. Do you use more traditional monitoring that will alert you somehow if a workflow fails, or a workflow hasn't run at all for X hrs/days?
- pela 7y agoAt Netflix, we rely on the workflow scheduler for such alerting and bundle in a layer of triggering mechanism (custom notifications and such)