6 ms·
Here are some key features: - Metaflow snapshots your code, data, and dependencies automatically in a content-addressed datastore, which is typically backed by
by vtuulos 7y ago
Here are some key features:
- Metaflow snapshots your code, data, and dependencies automatically in a content-addressed datastore, which is typically backed by S3, although local filesystem is supported too. This allows you to resume workflows, reproduce past results, and inspect anything about the workflow e.g. in a notebook. This is a core feature of Metaflow.
- Metaflow is designed to work well with a cloud backend. We support AWS today but technically other clouds could be supported too. There's quite a bit of engineering that has gone into building this integration. For instance, using the Metaflow's built-in S3 client, you can pull over 10Gbps, which is more than you can get with e.g. aws CLI today easily.
- We have spent time and effort in keeping the API surface area clean and highly usable. YMMV but it has been an appealing feature to many users this far.
Hope this makes sense!
- elamje 7y agoYour first bullet point should be highlighted on your project front page! I read over the page a couple of times, and couldn’t deduce that, and I think it’s a really enticing feature!
- savin-goyal 7y agoThanks. Feedback noted.
- missosoup 7y agoCan you compare and contrast with tools such as dask, dask-kubernetes, perfect[1]? [1] https://www.prefect.io/products/core https://www.prefect.io/products/core
- vtuulos 7y agoDask is great if you want to distribute your algorithms / data processing at a granular level. Metaflow is a bit more "meta" in a sense that we take your Python function as-is, which may use e.g. Tensorflow and PyTorch, and we execute it as an atomic unit on a container. This is useful, since you can use any existing ML libraries. I am not familiar with Prefect.
- tomrod 7y agoPrefect is built by the Airflow core devs after they took their initial learnings and built something new. It's a reasonable orchestration engine.
- seeravikiran 7y agoOur hope with metaflow is to make the transition to production schedulers like Airflow (and perhaps similar technologies) seamless once you write the DAG via the FlowSpec. The user doesn’t have to care about the conversion to YAML etc. So I would say metaflow works in tandem with existing schedulers.
- sandGorgon 7y agoThis is very interesting as a goal. Are you saying metaflow is a "Jupyter notebook for airflow developers" kind of a thing ?
- seeravikiran 7y agoI wouldn't exactly say that. Jupyter notebooks don't have an easy way to represent an arbitrary DAG. The flow is more linear and narrative like. That said, we do expect metaflow (with client API) to play very well with notebooks to support a narrative from a business use-case pov; which might be the end-goal of most ML workloads (hopefully). I would like to think of metaflow, as your workflow construct - hopefully making your life simpler with ML workloads when involving interactions with existing pieces of infrastructure (infra pieces - storage, compute, notebooks or other UI, http service hosting etc.; concepts - collaboration, versioning, archiving, dependency management)
- sandGorgon 7y agosorry- i meant notebook figuratively. so metaflow is the local dev version of workflow construct. and then export that to airflow/etc compatible format ? what workflow engine do you guys use and primarily support in metaflow ?
- khc 7y agoWhat does it mean to be able to pull over 10Gbps? With one object or many objects in the same prefix?
- seeravikiran 7y agoWith many objects under the same S3 bucket - say for a flow or a run (with many tasks).
- navinsylvester 7y ago> using the Metaflow's built-in S3 client, you can pull over 10Gbps, which is more than you can get with e.g. aws CLI today easily Can you please explain how you were able to better the performance of aws cli.
- vtuulos 7y agoyou need more connections than what a single AWS CLI process open to saturate network on a big box. You can achieve the same by doing some "xargs | aws cli" trickery but then error handling becomes harder. Our S3 client just handles multiple worker processes correctly with error handling.
- manojlds 7y agoCheck s4cmd - https://github.com/bloomreach/s4cmd https://github.com/bloomreach/s4cmd
- pbalau 7y agoSorry to be snarky, but after all this great job, I still have to put the db password in clear text, as an environment variable... Why not use secrets manager for this? It can even rotate the secret with not much headache. /edit: I could have a wrapper script that reads the secret and then os.execve()...
- savin-goyal 7y agoGood point. We will address it.