4 ms·

After going through a lot of marketing fluff, I landed on this useful page which explains Metaflow basics: https://docs.metaflow.org/metaflow/basics https://doc

After going through a lot of marketing fluff, I landed on this useful page which explains Metaflow basics: https://docs.metaflow.org/metaflow/basics https://docs.metaflow.org/metaflow/basics Here's my understanding: - It's a python library for creating & executing DAGs - Each node is a processing step & the results are stored after each step so you can restart failed workflows from where it failed - Tight integration with AWS ECS to run the whole DAG on cloud I don't know why their .org site oddly feels like a paid SaaS tool. Anyway, thank you Netflix for open sourcing Metaflow.

savin-goyal 7y ago
Addendum - You can mix and match what steps of the DAG run on the cloud.
seeravikiran 7y ago
I would also add - dependency management (certain degree of reproducibility) as a first class feature leveraging conda.
est 7y ago
looks like airflow with ML tools?
- thundergolfer 7y ago
  It provides a Python DAG building library like Airflow, but doesn't do Airflow's 'Operator ecosystem' thing. It also is very opinionated about dependency management (Conda-only) and is Python-only, where Airflow I think has operators to run arbitrary containers. So Metaflow is a non-starter I think if you don't want to exclusively use Python. Airflow also ships with built-in scheduler support (Celery?) or can run on K8s. Metaflow doesn't have this. Seems to rely on AWS Batch for production DAG execution. Airflow ships with a pretty rich UI. Metaflow seems to be anti-UI, and provides a novel Notebook-oriented workflow interaction model. Metaflow has pretty nice code artifact + params snapshotting functionality which is a core selling point. Airflow is not as supportive of this so it's harder to do reproducibility (I think). This is encapsulated by their "Datastore" model which can locally or in S3 persist flow code, config and data.
  - savin-goyal 7y ago
    Metaflow does come bundled with a scheduler that can place jobs on a variety of compute platforms (current release supports local on-instance and AWS batch). In terms of dependencies, we went with conda because of its traction in the data science community as well as excellent support for system packages. Our execution model also supports arbitrary docker containers (on AWS batch) where you can theoretically bake in your own dependencies. In terms of language support, we have bindings for R internally, that we plan to open source as well. I wouldn’t qualify metaflow as anti-UI. For model monitoring, we haven’t found a good enough UI that can handle the diversity of models and use cases we see internally, and believe that notebooks are an excellent visualisation medium that gives the power to the end user (data scientists) to craft dashboards as they see fit. For tracking the execution of production runs, we have historically relied on the UI of the scheduler itself (meson). We are exploring what a metaflow-specific UI might look like. As for comparisons with Airflow, it is an excellent production grade scheduler. Metaflow intends to solve a different problem of providing an excellent development and deployment experience for ML pipelines.
    - thundergolfer 7y ago
      Thanks for these clarifications. > Our execution model also supports arbitrary docker containers (on AWS batch) where you can theoretically bake in your own dependencies. That's fair, but it doesn't seem to be something encouraged by the framework, and that's fine. > I wouldn’t qualify metaflow as anti-UI. Maybe anti-UI is too strong yeah. I personally think your approach could be great. Looking forward to exploring it.
    - orbifold 7y ago
      Thanks for open sourcing this! This seems like precisely the kind of tool I‘ve been looking for. One thing that would be really great is support for HPC schedulers like SLURM. It seems relatively straightforward to add, so I might give it a shot myself.
      - seeravikiran 7y ago
        Happy to help either through our gitter chat or help@metaflow.org.
- seeravikiran 7y ago
  I guess (?) - minus the input spec being not YAML but more language native (pythonic for e.g.)
teekert 7y ago
So it's like Snakemake? [0] Snakemake can also control k8s clusters. Perhaps they are easier to set up with Metaflow? [0] https://snakemake.readthedocs.io/en/stable/ https://snakemake.readthedocs.io/en/stable/

savin-goyal 7y ago

Addendum - You can mix and match what steps of the DAG run on the cloud.

seeravikiran 7y ago

I would also add - dependency management (certain degree of reproducibility) as a first class feature leveraging conda.

est 7y ago

looks like airflow with ML tools?

thundergolfer 7y ago
It provides a Python DAG building library like Airflow, but doesn't do Airflow's 'Operator ecosystem' thing. It also is very opinionated about dependency management (Conda-only) and is Python-only, where Airflow I think has operators to run arbitrary containers. So Metaflow is a non-starter I think if you don't want to exclusively use Python. Airflow also ships with built-in scheduler support (Celery?) or can run on K8s. Metaflow doesn't have this. Seems to rely on AWS Batch for production DAG execution. Airflow ships with a pretty rich UI. Metaflow seems to be anti-UI, and provides a novel Notebook-oriented workflow interaction model. Metaflow has pretty nice code artifact + params snapshotting functionality which is a core selling point. Airflow is not as supportive of this so it's harder to do reproducibility (I think). This is encapsulated by their "Datastore" model which can locally or in S3 persist flow code, config and data.
- savin-goyal 7y ago
  Metaflow does come bundled with a scheduler that can place jobs on a variety of compute platforms (current release supports local on-instance and AWS batch). In terms of dependencies, we went with conda because of its traction in the data science community as well as excellent support for system packages. Our execution model also supports arbitrary docker containers (on AWS batch) where you can theoretically bake in your own dependencies. In terms of language support, we have bindings for R internally, that we plan to open source as well. I wouldn’t qualify metaflow as anti-UI. For model monitoring, we haven’t found a good enough UI that can handle the diversity of models and use cases we see internally, and believe that notebooks are an excellent visualisation medium that gives the power to the end user (data scientists) to craft dashboards as they see fit. For tracking the execution of production runs, we have historically relied on the UI of the scheduler itself (meson). We are exploring what a metaflow-specific UI might look like. As for comparisons with Airflow, it is an excellent production grade scheduler. Metaflow intends to solve a different problem of providing an excellent development and deployment experience for ML pipelines.
  - thundergolfer 7y ago
    Thanks for these clarifications. > Our execution model also supports arbitrary docker containers (on AWS batch) where you can theoretically bake in your own dependencies. That's fair, but it doesn't seem to be something encouraged by the framework, and that's fine. > I wouldn’t qualify metaflow as anti-UI. Maybe anti-UI is too strong yeah. I personally think your approach could be great. Looking forward to exploring it.
  - orbifold 7y ago
    Thanks for open sourcing this! This seems like precisely the kind of tool I‘ve been looking for. One thing that would be really great is support for HPC schedulers like SLURM. It seems relatively straightforward to add, so I might give it a shot myself.
    - seeravikiran 7y ago
      Happy to help either through our gitter chat or help@metaflow.org.
seeravikiran 7y ago
I guess (?) - minus the input spec being not YAML but more language native (pythonic for e.g.)

teekert 7y ago

So it's like Snakemake? [0] Snakemake can also control k8s clusters. Perhaps they are easier to set up with Metaflow? [0] https://snakemake.readthedocs.io/en/stable/ https://snakemake.readthedocs.io/en/stable/