3 ms·

This looks like a fantastically clean API for Python data and ML pipelines. Congratulations! It would be great to have a scheduler and monitoring UI that are e

This looks like a fantastically clean API for Python data and ML pipelines. Congratulations! It would be great to have a scheduler and monitoring UI that are equally lightweight.

vtuulos 7y ago
Metaflow comes with a built in scheduler. If your company has an existing scheduler, e.g. for ETL, you can translate Metaflow DAGs to the production scheduler automatically. This is what we do at Netflix. We could provide similar support e.g. for Airflow, if there's interest. For monitoring, we have relied on notebooks this far. Their flexibility and customizability is unbeatable. We might build some lightweight discovery / debugging UI later, if there's demand. We are a bit on the fence about it internally.
- thundergolfer 7y ago
  > Metaflow comes with a built in scheduler. This isn't a scheduler that will trigger 'flows' though is it? If I search "Scheduler" in your docs the top result is a roadmap item, and searching "Cron" turns up nothing. Is it true that right now that to run a DAG "every day at 3am UTC" requires an external service?
  - pela 7y ago
    Right now, yes.
    - thundergolfer 7y ago
      Thanks for confirmation.
- thundergolfer 7y ago
  > For monitoring, we have relied on notebooks this far. This is pretty interesting approach. Notebooks are a natural environment of Data Scientist and for monitoring they'd provide really good discoverability, flexibility, and interactivity. Do you use more traditional monitoring that will alert you somehow if a workflow fails, or a workflow hasn't run at all for X hrs/days?
  - pela 7y ago
    At Netflix, we rely on the workflow scheduler for such alerting and bundle in a layer of triggering mechanism (custom notifications and such)
jbenam 7y ago
Hey tristanz - we had similar needs and built up a lightweight scheduling, monitoring, and data flow UI. Originally for Airflow but we're excited about metaflow and will be integrating. Shoot me a note if interested to see what we built

This looks like a fantastically clean API for Python data and ML pipelines. Congratulations! It would be great to have a scheduler and monitoring UI that are equally lightweight.

vtuulos 7y ago

Metaflow comes with a built in scheduler. If your company has an existing scheduler, e.g. for ETL, you can translate Metaflow DAGs to the production scheduler automatically. This is what we do at Netflix. We could provide similar support e.g. for Airflow, if there's interest. For monitoring, we have relied on notebooks this far. Their flexibility and customizability is unbeatable. We might build some lightweight discovery / debugging UI later, if there's demand. We are a bit on the fence about it internally.

thundergolfer 7y ago
> Metaflow comes with a built in scheduler. This isn't a scheduler that will trigger 'flows' though is it? If I search "Scheduler" in your docs the top result is a roadmap item, and searching "Cron" turns up nothing. Is it true that right now that to run a DAG "every day at 3am UTC" requires an external service?
- pela 7y ago
  Right now, yes.
  - thundergolfer 7y ago
    Thanks for confirmation.
thundergolfer 7y ago
> For monitoring, we have relied on notebooks this far. This is pretty interesting approach. Notebooks are a natural environment of Data Scientist and for monitoring they'd provide really good discoverability, flexibility, and interactivity. Do you use more traditional monitoring that will alert you somehow if a workflow fails, or a workflow hasn't run at all for X hrs/days?
- pela 7y ago
  At Netflix, we rely on the workflow scheduler for such alerting and bundle in a layer of triggering mechanism (custom notifications and such)

jbenam 7y ago

Hey tristanz - we had similar needs and built up a lightweight scheduling, monitoring, and data flow UI. Originally for Airflow but we're excited about metaflow and will be integrating. Shoot me a note if interested to see what we built