savin-goyal

8 ms·

31.
Retrieval-Augmented Generation: How to Use Your Data to Guide LLMs(outerbounds.com)
4 points by savin-goyal 3y ago|0 comments
32.
Real-time infrastructure tooling for data scientists(quix.io)
1 points by savin-goyal 3y ago|0 comments
33.
by savin-goyal 4y ago
Metaflow passes state using an object store (s3, azure blob store, etc.) even within Airflow - short circuiting Airflow's xcom machinery. But agreed, Airflow presents many more scalability challenges - this integration addresses a few
34.
by savin-goyal 4y ago
Outerbounds | Systems dev | SF / Remote | Full-time Outerbounds was founded recently to commercialize Metaflow, an open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects
35.
Open-Sourcing Metaflow, a Human-Centric Framework for Data Science(netflixtechblog.com)
3 points by savin-goyal 4y ago|0 comments
36.
by savin-goyal 4y ago
Metaflow provides a similar concept to interface with Step Functions and Argo Workflows in Python - https://docs.metaflow.org/going-to-production-with-metaflow/...
37.
Human-Centric Data Science on Kubernetes with Netflix Metaflow(blog.argoproj.io)
7 points by savin-goyal 4y ago|0 comments
38.
by savin-goyal 5y ago
A single machine can take you remarkably far these days, given the availability of high RAM/Disk/CPU machines in the cloud.
39.
Ransomware Attacker Offers Employees a Cut(beta.darkreading.com)
2 points by savin-goyal 5y ago|0 comments
40.
Machine Learning at CNN(medium.com)
23 points by savin-goyal 5y ago|10 comments
41.
How big should a microservice be(bovon.org)
4 points by savin-goyal 5y ago|2 comments
42.
by savin-goyal 6y ago
I think another very important piece of the puzzle is *when* to make the transition from a monolith service to a microservice stack. I have seen many startups getting distracted by committing to a microservices transition rather too early i
43.
by savin-goyal 6y ago
Metaflow was built to assist in both developing ML models and deploying/managing them in production. AFAIK, TFX is focused on the deployment story of ML pipelines. https://docs.metaflow.org/introduction/what-is-met
44.
by savin-goyal 6y ago
Give it a few minutes :)
45.
by savin-goyal 6y ago
Hi! Metaflow ships with a CloudFormation template for AWS that automates the set-up of a blob store (S3), compute environment (Batch), metadata tracking service (RDS), orchestrator (Step-Functions) notebooks (Sagemaker) and all the necessar
46.
by savin-goyal 6y ago
Take a look at metaflow.org/sandbox if you want to test drive Metaflow.
47.
by savin-goyal 7y ago
https://news.ycombinator.com/item?id=21704306
48.
by savin-goyal 7y ago
Fixed
49.
by savin-goyal 7y ago
I am not familiar with Meltano, sorry.
50.
by savin-goyal 7y ago
You’re welcome! :)
51.
by savin-goyal 7y ago
Good point. We will address it.
52.
by savin-goyal 7y ago
All of our integrations are driven by plugins. We are exploring our roadmap with regards to integrations with other clouds.
53.
by savin-goyal 7y ago
Yes, you can use any python library inside a metaflow step.
54.
by savin-goyal 7y ago
Metaflow does come bundled with a scheduler that can place jobs on a variety of compute platforms (current release supports local on-instance and AWS batch). In terms of dependencies, we went with conda because of its traction in the data s
55.
by savin-goyal 7y ago
Addendum - You can mix and match what steps of the DAG run on the cloud.
56.
by savin-goyal 7y ago
Thanks. Feedback noted.
57.
by savin-goyal 7y ago
I am not familiar with them.
58.
by savin-goyal 7y ago
We erred on the side of simplicity to keep things manageable for our users.
59.
by savin-goyal 7y ago
Hi omarhaneef, We don't intend to compete with Tensorflow, PyTorch, SKLearn. What we offer is a way to iterate and productionize your models written using any of the aforementioned libraries (and more). https://docs.metaflow