6 ms·

hey, I'm one of the authors of Metaflow. Happy to answer any questions! Netflix has been using Metaflow internally for about two years, so we have many war stor

hey, I'm one of the authors of Metaflow. Happy to answer any questions! Netflix has been using Metaflow internally for about two years, so we have many war stories :)

deleted 7y ago
[deleted]
bhtucker 7y ago
The centralized DAG scheduler seems like a pretty important part. How much will not having Meson hamper the usability?
- pela 7y ago
  Hi bhtucker, We do have plans to integrate with production-grade schedulers in the very near future. https://github.com/Netflix/metaflow/issues/2 https://github.com/Netflix/metaflow/issues/2
- vtuulos 7y ago
  note that Metaflow comes with a built-in DAG scheduler, so it is perfectly usable without a centralized scheduler like Meson. Integrations to Airflow or StepFunctions are on the roadmap, depending on what seems to resonate with people outside Netflix.
ageofwant 7y ago
G'day, seems like an cool tool, thanks - the links to the github tuts are currently broken... I'm of the opinion that adopting some standard DAG meta format for data science may make a positive impact on the reproduceability issues we have in science generally. So its good to see the idea has real world merit as well.
- pela 7y ago
  Hi ageofwant, Thanks for taking a look at Metaflow. We will fix these links. Can you point us to the offending links?
  - pela 7y ago
    I have updated the links.
    - LegitShady 7y ago
      Pray you do not update them further?
omarhaneef 7y ago
Thanks for open sourcing this. Can you say a little about which niche this would occupy, and what the motivation is? Is it intended to compete with Tensorflow and Pytorch or to be an industrial strength version of SKlearn. I looked through the tutorial on my mobile and the answer was not immediately clear. Is the benefit that it auto scales on AWS without having to think through the infrastructure?
- savin-goyal 7y ago
  Hi omarhaneef, We don't intend to compete with Tensorflow, PyTorch, SKLearn. What we offer is a way to iterate and productionize your models written using any of the aforementioned libraries (and more). https://docs.metaflow.org/introduction/what-is-metaflow https://docs.metaflow.org/introduction/what-is-metaflow contains further elaboration of our philosophy. Auto-scaling infrastructure is one piece of the puzzle, and Metaflow goes beyond that offering a comprehensive solution for model management.
kprybol 7y ago
Does this do experiment tracking similar to MLflow? I’m trying to figure where these two overlap and where they diverge.
- vtuulos 7y ago
  yep, Metaflow tracks everything: Your code, dependencies, and the internal state of the workflow automatically. A big difference between Metaflow and other workflow frameworks for ML is that Metaflow doesn't only execute your DAG, it helps you to design and implement the code that runs inside the DAG. Many frameworks leave these details to the data scientist to decide.
jmpeax 7y ago
Why are my Netflix suggestions and % so bad? Because I like "There Will Be Blood" doesn't mean I'm going to like "Ridiculous 6" just because both a westerns. Or, because I like "Fight Club" doesn't mean I'm going to like "By The Sea" just because both have Brad Pitt in them.
- thundergolfer 7y ago
  This is pretty off-topic, and you're asking an infrastructure engineer who is not going to be able to answer this.
  - Supermancho 7y ago
    No, it is not offtopic in the slightest...especially in response to an "author". The underlying questions remain obvious. Is metaflow not used in netflix suggestions? What utility does it offer that other tooling doesnt or at least how has netflix extracted value? These are interesting questions.
    - thundergolfer 7y ago
      I replied to someone complaining about their Netflix movie recommendations being bad. Did you think I was replying to you?
ordx 7y ago
Thank you for sharing this. Would this be useful for me if I only need the deployment management part? Don't really need to track experiments, just looking for an easy way to deploy my models to Fargate.
- pela 7y ago
  Yes indeed. You can access your model via the Metaflow client inside your service - https://docs.metaflow.org/metaflow/client https://docs.metaflow.org/metaflow/client. We are looking into releasing the hosting component on Metaflow in the future - https://github.com/Netflix/metaflow/issues/3 https://github.com/Netflix/metaflow/issues/3
zmjjmz 7y ago
Hey I meant to track one of y'all down at the MLOps conference, but didn't get the chance. I've built a very shitty version of a cached-execution DAG thing internally, and one of the design decisions I made was to have it so that parent nodes in the DAG don't need to know anything about child nodes. This allows for larger DAG builders to be more easily subclassed. MetaFlow doesn't do that -- instead each 'step' has to know what to call next, which means that if I wanted to subclass e.g. the MovieStatsFlow in [here](https://github.com/Netflix/metaflow/blob/master/metaflow/tutorials/02-statistics/stats.py#L57 https://github.com/Netflix/metaflow/blob/master/metaflow/tut...) and say, add some sort of input pre-processing before the compute_statistics call, I'd essentially end up having to either override what compute_statistics does to not match its name _or_ copy-past e that first step just to replace that last line. I'm sure this design decision was considered and/or that use-case doesn't come up a lot at Netflix (although I've encountered that a lot), or maybe I'm missing something very obvious, but I'd love to hear your thoughts on that.
- savin-goyal 7y ago
  We erred on the side of simplicity to keep things manageable for our users.
  - beshrkayali 7y ago
    That seems more complicated.
faizshah 7y ago
I have been looking for something exactly like this, but I use GCP not AWS. Is there a way to deploy outside of AWS? What would be involved in getting it to work with a different cloud?
- savin-goyal 7y ago
  All of our integrations are driven by plugins. We are exploring our roadmap with regards to integrations with other clouds.
rgudic 7y ago
Hi Ville, I cannot express how happy I am that you open-sourced Metaflow. I have three questions: 1) Do you know ETA of R API release? 2) Would you recommend using both Metaflow nad MLFlow in projects? Could you please explain why yes or why no :) 3) Do you plan to release integration with Spark/Yarn? Thanks in advance, Roko

hey, I'm one of the authors of Metaflow. Happy to answer any questions! Netflix has been using Metaflow internally for about two years, so we have many war stories :)

deleted 7y ago

[deleted]

bhtucker 7y ago

The centralized DAG scheduler seems like a pretty important part. How much will not having Meson hamper the usability?

pela 7y ago
Hi bhtucker, We do have plans to integrate with production-grade schedulers in the very near future. https://github.com/Netflix/metaflow/issues/2 https://github.com/Netflix/metaflow/issues/2
vtuulos 7y ago
note that Metaflow comes with a built-in DAG scheduler, so it is perfectly usable without a centralized scheduler like Meson. Integrations to Airflow or StepFunctions are on the roadmap, depending on what seems to resonate with people outside Netflix.

ageofwant 7y ago

G'day, seems like an cool tool, thanks - the links to the github tuts are currently broken... I'm of the opinion that adopting some standard DAG meta format for data science may make a positive impact on the reproduceability issues we have in science generally. So its good to see the idea has real world merit as well.

pela 7y ago
Hi ageofwant, Thanks for taking a look at Metaflow. We will fix these links. Can you point us to the offending links?
- pela 7y ago
  I have updated the links.
  - LegitShady 7y ago
    Pray you do not update them further?

omarhaneef 7y ago

Thanks for open sourcing this. Can you say a little about which niche this would occupy, and what the motivation is? Is it intended to compete with Tensorflow and Pytorch or to be an industrial strength version of SKlearn. I looked through the tutorial on my mobile and the answer was not immediately clear. Is the benefit that it auto scales on AWS without having to think through the infrastructure?

savin-goyal 7y ago
Hi omarhaneef, We don't intend to compete with Tensorflow, PyTorch, SKLearn. What we offer is a way to iterate and productionize your models written using any of the aforementioned libraries (and more). https://docs.metaflow.org/introduction/what-is-metaflow https://docs.metaflow.org/introduction/what-is-metaflow contains further elaboration of our philosophy. Auto-scaling infrastructure is one piece of the puzzle, and Metaflow goes beyond that offering a comprehensive solution for model management.

kprybol 7y ago

Does this do experiment tracking similar to MLflow? I’m trying to figure where these two overlap and where they diverge.

vtuulos 7y ago
yep, Metaflow tracks everything: Your code, dependencies, and the internal state of the workflow automatically. A big difference between Metaflow and other workflow frameworks for ML is that Metaflow doesn't only execute your DAG, it helps you to design and implement the code that runs inside the DAG. Many frameworks leave these details to the data scientist to decide.

jmpeax 7y ago

Why are my Netflix suggestions and % so bad? Because I like "There Will Be Blood" doesn't mean I'm going to like "Ridiculous 6" just because both a westerns. Or, because I like "Fight Club" doesn't mean I'm going to like "By The Sea" just because both have Brad Pitt in them.

thundergolfer 7y ago
This is pretty off-topic, and you're asking an infrastructure engineer who is not going to be able to answer this.
- Supermancho 7y ago
  No, it is not offtopic in the slightest...especially in response to an "author". The underlying questions remain obvious. Is metaflow not used in netflix suggestions? What utility does it offer that other tooling doesnt or at least how has netflix extracted value? These are interesting questions.
  - thundergolfer 7y ago
    I replied to someone complaining about their Netflix movie recommendations being bad. Did you think I was replying to you?

ordx 7y ago

Thank you for sharing this. Would this be useful for me if I only need the deployment management part? Don't really need to track experiments, just looking for an easy way to deploy my models to Fargate.

pela 7y ago
Yes indeed. You can access your model via the Metaflow client inside your service - https://docs.metaflow.org/metaflow/client https://docs.metaflow.org/metaflow/client. We are looking into releasing the hosting component on Metaflow in the future - https://github.com/Netflix/metaflow/issues/3 https://github.com/Netflix/metaflow/issues/3

zmjjmz 7y ago

Hey I meant to track one of y'all down at the MLOps conference, but didn't get the chance. I've built a very shitty version of a cached-execution DAG thing internally, and one of the design decisions I made was to have it so that parent nodes in the DAG don't need to know anything about child nodes. This allows for larger DAG builders to be more easily subclassed. MetaFlow doesn't do that -- instead each 'step' has to know what to call next, which means that if I wanted to subclass e.g. the MovieStatsFlow in [here](https://github.com/Netflix/metaflow/blob/master/metaflow/tutorials/02-statistics/stats.py#L57 https://github.com/Netflix/metaflow/blob/master/metaflow/tut...) and say, add some sort of input pre-processing before the compute_statistics call, I'd essentially end up having to either override what compute_statistics does to not match its name _or_ copy-past e that first step just to replace that last line. I'm sure this design decision was considered and/or that use-case doesn't come up a lot at Netflix (although I've encountered that a lot), or maybe I'm missing something very obvious, but I'd love to hear your thoughts on that.

savin-goyal 7y ago
We erred on the side of simplicity to keep things manageable for our users.
- beshrkayali 7y ago
  That seems more complicated.

faizshah 7y ago

I have been looking for something exactly like this, but I use GCP not AWS. Is there a way to deploy outside of AWS? What would be involved in getting it to work with a different cloud?

savin-goyal 7y ago
All of our integrations are driven by plugins. We are exploring our roadmap with regards to integrations with other clouds.

rgudic 7y ago

Hi Ville, I cannot express how happy I am that you open-sourced Metaflow. I have three questions: 1) Do you know ETA of R API release? 2) Would you recommend using both Metaflow nad MLFlow in projects? Could you please explain why yes or why no :) 3) Do you plan to release integration with Spark/Yarn? Thanks in advance, Roko