3 ms·

I went to the GitHub page. The descriptions of the service seem redundant to what cloud providers offer today. I looked at the documentation and it lacks conc

I went to the GitHub page. The descriptions of the service seem redundant to what cloud providers offer today. I looked at the documentation and it lacks concrete examples for implementation flows. Seems like something new to learn, an added layer on top of existing workflows, with no obvious benefit.

manojlds 1y ago
It's an old project from before the current AI buzz and I rejected this when I looked at it few years back as well with similar reasons. My opinion about Netflix OSS has been pretty low as well.
datadrivenangel 1y ago
All the cloud providers have some hosted / custom version of an AI/ML deployment and training system. Good enough to use, janky enough to probably not meet all your needs if you're serious.
- lazarus01 1y ago
  I use google cloud for ML. AWS has a similar offering. I find google is purpose built for ml and provides tons of resources with excellent documentation. AWS feels like driving a double decker bus, very big and clunky, compared to google, which is a luxury sedan, that is quite comfortable to take you where you’re going.
vibecodemaster 1y ago
> redundant to what cloud providers offer today It may look redundant on the surface, but those cloud services are infrastructure primitives (compute, storage, orchestration). Metaflow sits one layer higher, giving you a data/model centric API that orchestrates and versions the entire workflow (code, data, environment, and lineage) while delegating the low‑level plumbing to whatever cloud provider you choose. That higher‑level abstraction is what lets the same Python flow run untouched on a laptop today and a K8s GPU cluster tomorrow. > Adds an extra layer to learn I would argue that it removes layers: you write plain Python functions, tag them as steps, and Metaflow handles scheduling, data movement, retry logic, versioning, and caching. You no longer glue together five different SDKs (batch + orchestration + storage + secrets + lineage). > lacks concrete examples for implementation flows there are examples in the tutorials: https://docs.outerbounds.com/intro-tutorial-season-3-overview/ https://docs.outerbounds.com/intro-tutorial-season-3-overvie... > with no obvious benefit There are benefits, but perhaps they're not immediately obvious: 1) Separation of what vs. how: declare the workflow once; toggle @resources(cpu=4,gpu=1) to move from dev to a GPU cluster—no YAML rewrites. 2) Reproducibility & lineage: every run immutably stores code, data hashes, and parameters so you can reproduce any past model or report with flow resume --run-id. 3) Built‑in data artifacts: pass or version GB‑scale objects between steps without manually wiring S3 paths or serialization logic.

manojlds 1y ago

It's an old project from before the current AI buzz and I rejected this when I looked at it few years back as well with similar reasons. My opinion about Netflix OSS has been pretty low as well.

datadrivenangel 1y ago

All the cloud providers have some hosted / custom version of an AI/ML deployment and training system. Good enough to use, janky enough to probably not meet all your needs if you're serious.

lazarus01 1y ago
I use google cloud for ML. AWS has a similar offering. I find google is purpose built for ml and provides tons of resources with excellent documentation. AWS feels like driving a double decker bus, very big and clunky, compared to google, which is a luxury sedan, that is quite comfortable to take you where you’re going.

vibecodemaster 1y ago

> redundant to what cloud providers offer today It may look redundant on the surface, but those cloud services are infrastructure primitives (compute, storage, orchestration). Metaflow sits one layer higher, giving you a data/model centric API that orchestrates and versions the entire workflow (code, data, environment, and lineage) while delegating the low‑level plumbing to whatever cloud provider you choose. That higher‑level abstraction is what lets the same Python flow run untouched on a laptop today and a K8s GPU cluster tomorrow. > Adds an extra layer to learn I would argue that it removes layers: you write plain Python functions, tag them as steps, and Metaflow handles scheduling, data movement, retry logic, versioning, and caching. You no longer glue together five different SDKs (batch + orchestration + storage + secrets + lineage). > lacks concrete examples for implementation flows there are examples in the tutorials: https://docs.outerbounds.com/intro-tutorial-season-3-overview/ https://docs.outerbounds.com/intro-tutorial-season-3-overvie... > with no obvious benefit There are benefits, but perhaps they're not immediately obvious: 1) Separation of what vs. how: declare the workflow once; toggle @resources(cpu=4,gpu=1) to move from dev to a GPU cluster—no YAML rewrites. 2) Reproducibility & lineage: every run immutably stores code, data hashes, and parameters so you can reproduce any past model or report with flow resume --run-id. 3) Built‑in data artifacts: pass or version GB‑scale objects between steps without manually wiring S3 paths or serialization logic.