6 ms·
Cortex: Deploy machine learning models in production
- kevinmershon 7y agoUnfortunately, a somewhat popular Clojure library for machine learning on GitHub is also called Cortex, because this is going to make discussing machine learning APIs in the context of Clojure that much more confusing.
- brennebeck 7y agoCouldn’t you just search ‘clojure cortex’? As this isn’t actually clojure?
- ChefboyOG 7y agoIt also looks like the last release for the Clojure library was in 2017
- waz0wski 7y agothere's also a prometheus storage backend called Cortex https://github.com/cortexproject/cortex https://github.com/cortexproject/cortex
- notus 7y agoLast commit was like 2 years ago...
- deleted 7y ago[deleted]
- bermanoid 7y agoAnd I imagine many more machine learning tools will take the same name in the years to come, since it's about the most obvious one you could think of other than "brain". Whatever is popular will survive...
- mzanchi 7y agoCalling it an alternative to SageMaker might be a bit misleading, as SageMaker is also a platform for training the models in automatically allocated EC2 resources, even on spot instances.
- dang 7y agoWe've changed the title from "Cortex: An open source alternative to SageMaker" to the page's own title, as the HN guidelines request. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- deliahu 7y agoCortex contributor here - you're right, I would say we can be compared to SageMaker model deployment. We are currently working on supporting spot instances for serving, and training is on our roadmap.
- manojlds 7y agoSagemaker has notebooks, training and serving. This seems to be only about the serving.
- lettergram 7y agoWas discussed three months ago: https://news.ycombinator.com/item?id=20579166 https://news.ycombinator.com/item?id=20579166
- punnerud 7y agoSeems to work with: Pytorch, TensorFlow, Keras, XGBoost, sklearn
- ospillinger 7y agoYes, Cortex uses ONNX Runtime (https://github.com/microsoft/onnxruntime https://github.com/microsoft/onnxruntime) under the hood so any model that can be exported to ONNX can be deployed.
- solidasparagus 7y agoIs it only able to handle ONNX models? That's a pretty massive limitation compared to a hosted SageMaker endpoint.
- vishalbollu 7y agoContributor here - Cortex supports Tensorflow saved models in addition to ONNX. PyTorch support is on the roadmap. Do you have specific frameworks in mind that you would like Cortex to support?
- solidasparagus 7y agoPerfect. Nothing in particular other than TF.
- kuu 7y agoHow does this work under the hood? Is the model loaded every time it receives a request? Is it run in a docker or a lambda? How does it work after "uploading it" to amazon?
- ospillinger 7y agoEach model is loaded into a Docker container, along with any Python packages and request handling code. The cluster runs on EKS on your AWS account. Cortex takes the declarative configuration from 'cortex.yaml' and creates it every time you run 'cortex deploy' so the containers don’t change unless you run 'cortex deploy' again with updated configuration. This post goes into more detail about some of our design decisions: https://towardsdatascience.com/inference-at-scale-49bc222b3ad1 https://towardsdatascience.com/inference-at-scale-49bc222b3a...
- kuu 7y agoThank you!
- oli5679 7y agoIf your model can be exported as PMML, this is really nice. Fast, minimalist, battle-tested and with very clean API. When I've tested, it's up to 10x faster than Flask + serialised model object and uses far less CPU resources. Plays nicely with lightgbm and Xgboost. https://github.com/openscoring/openscoring https://github.com/openscoring/openscoring
- isubasinghe 7y agoThis is basically my startup idea that I worked on for a while now (https://aiscalr.isub.dev https://aiscalr.isub.dev) Looks like I am going to have to scrap that entire project now, seems pointless to keep working on it given how similar this is.
- sixhobbits 7y agoSimilarity should be taken as validation, not a negative thing at all.
- ovi256 7y agoYou should do customer development and find people willing to pay for your product. If they're willing to pay, they'll even tell you why they can't use the open source tool.
- TaupeRanger 7y agoStop naming things single word neuroscience terms. There are like 50 projects called "Cortex".
- ecnahc515 7y agoNot to be confused with weaveworks/CNCF Cortex project for high scale Prometheus monitoring https://github.com/cortexproject/cortex https://github.com/cortexproject/cortex.
- sails 7y agoHow does this compare to MLflow [0]? Considering MLflow has a few components, I suppose you are building something closer to MLflow Models? How do they compare? [0] https://mlflow.org/docs/latest/index.html https://mlflow.org/docs/latest/index.html
- ospillinger 7y agoFrom the MLflow Models docs: "An MLflow Model is a standard format for packaging machine learning models that can be used in a variety of downstream tools—for example, real-time serving through a REST API or batch inference on Apache Spark. The format defines a convention that lets you save a model in different “flavors” that can be understood by different downstream tools." Cortex is what they are referring to as a downstream tool for real-time serving through a REST API. In other words, MLflow helps with model management and packaging, whereas Cortex is a platform for running real-time inference at scale. We are working on supporting more model packaging formats and I think it's a good idea to support the MLflow format as well.