seeravikiran

5 ms·

1. 1.▲
   by seeravikiran 5y ago

   What were some of the pain points you face(d) - looking back at your Metaflow adoption? Disclaimer: I work in Netflix ML Platform that helped open-source Metaflow originally.
2. 2.▲
   by seeravikiran 7y ago

   Happy to help either through our gitter chat or help@metaflow.org.
3. 3.▲
   by seeravikiran 7y ago

   Thanks for reporting it. We ll fix it. Sorry for the inconvenience.
4. 4.▲
   by seeravikiran 7y ago

   Thanks. Let us know how you like the prototyping -> scaling out & up journey.
5. 5.▲
   by seeravikiran 7y ago

   I wouldn't exactly say that. Jupyter notebooks don't have an easy way to represent an arbitrary DAG. The flow is more linear and narrative like. That said, we do expect metaflow (with client API) to play very well with notebooks t
6. 6.▲
   by seeravikiran 7y ago

   Thanks for sharing the context. Hopefully we can have a (fast) follow up with Kube integration depending on demand.
7. 7.▲
   by seeravikiran 7y ago

   Yes - you should be able to use dask the way you say. Your first part of the understanding matches my expectation too. Dask single box parallelism achieved by multi processing - akin to parallel map. And distributed compute is achieved by s
8. 8.▲
   by seeravikiran 7y ago

   I guess (?) - minus the input spec being not YAML but more language native (pythonic for e.g.)
9. 9.▲
   by seeravikiran 7y ago

   Yes - that’s our thinking too. Compilers finding your typos for variable names seems helpful for user productivity.
10. 10.▲
    by seeravikiran 7y ago

    Thanks for pinging on this. re: Kubeflow - imho it is quite coupled to Kubernetes. We don’t intend to be tied to a specific compute substrate even though the first launch is with AWS. We do follow a plugin architecture - so I’m hoping Kube
11. 11.▲
    by seeravikiran 7y ago

    I would also add - dependency management (certain degree of reproducibility) as a first class feature leveraging conda.
12. 12.▲
    by seeravikiran 7y ago

    Our hope with metaflow is to make the transition to production schedulers like Airflow (and perhaps similar technologies) seamless once you write the DAG via the FlowSpec. The user doesn’t have to care about the conversion to YAML etc. So I
13. 13.▲
    by seeravikiran 7y ago

    With many objects under the same S3 bucket - say for a flow or a run (with many tasks).
14. 14.▲
    by seeravikiran 7y ago

    1. Metaflow should best help when there is an element of collaboration - so small to medium team of data scientists. Collaborating with your self is also another scenario when Metaflow can be useful since it takes care of versioning and arc