4 ms·

Data scientist here, stuck in the Dark Ages of "deploying" my models by writing bespoke Python apps that run on some kind of cloud container host like ECS. Dump

Data scientist here, stuck in the Dark Ages of "deploying" my models by writing bespoke Python apps that run on some kind of cloud container host like ECS. Dump the outputs to blob storage and slurp them back into the data warehouse nightly using Airflow. Lots of manual fussing around. What the heck are all these ML and data platforms, how do they benefit me, and how do I evaluate the gazillion options that seem to be out there? For example, I recently came across DStack (https://dstack.ai/ https://dstack.ai/) and have had an open browser tab sitting around waiting for me to figure out WTF it even does. DAGWorks seems like it does something similar. Is that true? Are these tools even comparable? How would I choose one or the other? Is there overlap with MLFlow?

sampo 4y ago
> bespoke Python apps that run on some kind of cloud container host like ECS. For this, you could use for example https://flyte.org/ https://flyte.org/, if you have platform engineers that could set it up for you. But I don't think their website makes a good job of explaining what it is and what it does.
- krawczstef 4y ago
  yep some of these ML platform tools only make sense once you reach a certain scale/set of problems. For flyte, it was created because Lyft had a very heterogeneous set of tools one could build a ML pipeline with. So they made it really easy to integrate and importantly serialize, data between systems. E.g. sql -> python -> spark -> python. But if all your data fits in memory, or you only one system, you might not be a great fit for Flyte. What I try to do, is understand who created the platform, and understand the environment of that company. Then that will give you a better idea of whether it makes sense for you or not.
  - kumare3 4y ago
    I do agree it makes sense fornscale, but if your data fits in memory Flyte native constructs shine. For example it will ensure your data is stored / serialized correctly. Allows you to use polars, vaex, duckdb etc. tbh I am a huge proponent of vertical scaling till you can get the mileage. It also supports - gpu allocation, spot instances and collaboration across multiple users. I do not think it is a wrong choice if you feel your complexity will grow. PS. I am a maintainer of the project
    - krawczstef 4y ago
      Yep - it's on my TODO to get https://github.com/flyteorg/flyte/issues/2627 https://github.com/flyteorg/flyte/issues/2627 over the line!
- kumare3 4y ago
  Thank you for sharing. If you do not have platform engineers look at union.ai. They offer a manager version of Flyte
elijahbenizzy 4y ago
TL;DR -- ML platforms solve for a million different problems, and most people don't have all (or any) of them. Hamilton is a pretty simple way of organizing code, so it can plug in with a bunch of different approaches (and that's why we think its general purpose). > Data scientist here, stuck in the Dark Ages of "deploying" my models by writing bespoke Python apps that run on some kind of cloud container host like ECS. Dump the outputs to blob storage and slurp them back into the data warehouse nightly using Airflow. Lots of manual fussing around. Oh, man, been there! So, first, I want to say there are a lot of ML/data platforms -- largely because there are so many problems to solve, and they're not one-size-fits-all solutions. > What the heck are all these ML and data platforms, how do they benefit me, and how do I evaluate the gazillion options that seem to be out there? You probably don't need all of them, or all that many. As in everything, it depends on your pain-points. Given that you have a lot of manual fussing around, you probably want something to reduce it. We've found airflow to be painful, but a lot of dev teams/DS have airflow already integrated into their platform, so we wanted to build something that allows data scientists to plug into it. So for people who don't like airflow, the idea is that you could express your dataflow in Hamilton and DAGWorks can ship it to airflow (or any other orchestration system). > For example, I recently came across DStack (https://dstack.ai/ https://dstack.ai/) and have had an open browser tab sitting around waiting for me to figure out WTF it even does. DAGWorks seems like it does something similar. Is that true? Are these tools even comparable? How would I choose one or the other? Is there overlap with MLFlow? DStack is definitely a different approach, similar space. Hamilton is organized around python functions and dstack is more of a high-level workflow spec (reminds me of something we had at my old company). So Hamilton can model the "micro" of your workflow, whereas DStack models the "macro" -- managing artifacts. DStack could easily run Hamilton functions. What nothing we've found out there do (except perhaps kedro) is model the "micro" -- E.G. the specific fine-grained dependencies so you can take a look at your code and figure out how exactly it works. Re: MLFlow -- DAGWorks + MLFlow are pretty natural connectors. Hamilton functions can produce a model that DAGWorks would be able to save to mlflow. DAGWorks is more on the data transform side, and doesn't explicitly say how to represent a model.
- deleted 4y ago
  [deleted]
cheptsov 4y ago
Hey, the founder of dstack here. If I put it shortly, dstack allows you to define ML workflows as code and run them either locally or remotely (e.g. in a configured cloud). ML workflows here mean anything that you may want to do when you're developing a model - prepping data, training or finetuning a model, etc. The value - basically it automates running your workflows, without being dependant on any particular vendor. At the same time, you don't have to rewrite your Python scripts to use a particular API (because dstack is using YAML). We aim to build the most easy tool to run ML workflows - without making you use the UI of any vendor, or hustling with Kubernetes, custom Docker images, etc. MLFlow doesn't do what dstack does (automatic infrastructure provisioning) - unless you use Databricks.
- krawczstef 4y ago
  Nice! We had a similar abstraction at Stitch Fix.

sampo 4y ago

> bespoke Python apps that run on some kind of cloud container host like ECS. For this, you could use for example https://flyte.org/ https://flyte.org/, if you have platform engineers that could set it up for you. But I don't think their website makes a good job of explaining what it is and what it does.

krawczstef 4y ago
yep some of these ML platform tools only make sense once you reach a certain scale/set of problems. For flyte, it was created because Lyft had a very heterogeneous set of tools one could build a ML pipeline with. So they made it really easy to integrate and importantly serialize, data between systems. E.g. sql -> python -> spark -> python. But if all your data fits in memory, or you only one system, you might not be a great fit for Flyte. What I try to do, is understand who created the platform, and understand the environment of that company. Then that will give you a better idea of whether it makes sense for you or not.
- kumare3 4y ago
  I do agree it makes sense fornscale, but if your data fits in memory Flyte native constructs shine. For example it will ensure your data is stored / serialized correctly. Allows you to use polars, vaex, duckdb etc. tbh I am a huge proponent of vertical scaling till you can get the mileage. It also supports - gpu allocation, spot instances and collaboration across multiple users. I do not think it is a wrong choice if you feel your complexity will grow. PS. I am a maintainer of the project
  - krawczstef 4y ago
    Yep - it's on my TODO to get https://github.com/flyteorg/flyte/issues/2627 https://github.com/flyteorg/flyte/issues/2627 over the line!
kumare3 4y ago
Thank you for sharing. If you do not have platform engineers look at union.ai. They offer a manager version of Flyte

elijahbenizzy 4y ago

TL;DR -- ML platforms solve for a million different problems, and most people don't have all (or any) of them. Hamilton is a pretty simple way of organizing code, so it can plug in with a bunch of different approaches (and that's why we think its general purpose). > Data scientist here, stuck in the Dark Ages of "deploying" my models by writing bespoke Python apps that run on some kind of cloud container host like ECS. Dump the outputs to blob storage and slurp them back into the data warehouse nightly using Airflow. Lots of manual fussing around. Oh, man, been there! So, first, I want to say there are a lot of ML/data platforms -- largely because there are so many problems to solve, and they're not one-size-fits-all solutions. > What the heck are all these ML and data platforms, how do they benefit me, and how do I evaluate the gazillion options that seem to be out there? You probably don't need all of them, or all that many. As in everything, it depends on your pain-points. Given that you have a lot of manual fussing around, you probably want something to reduce it. We've found airflow to be painful, but a lot of dev teams/DS have airflow already integrated into their platform, so we wanted to build something that allows data scientists to plug into it. So for people who don't like airflow, the idea is that you could express your dataflow in Hamilton and DAGWorks can ship it to airflow (or any other orchestration system). > For example, I recently came across DStack (https://dstack.ai/ https://dstack.ai/) and have had an open browser tab sitting around waiting for me to figure out WTF it even does. DAGWorks seems like it does something similar. Is that true? Are these tools even comparable? How would I choose one or the other? Is there overlap with MLFlow? DStack is definitely a different approach, similar space. Hamilton is organized around python functions and dstack is more of a high-level workflow spec (reminds me of something we had at my old company). So Hamilton can model the "micro" of your workflow, whereas DStack models the "macro" -- managing artifacts. DStack could easily run Hamilton functions. What nothing we've found out there do (except perhaps kedro) is model the "micro" -- E.G. the specific fine-grained dependencies so you can take a look at your code and figure out how exactly it works. Re: MLFlow -- DAGWorks + MLFlow are pretty natural connectors. Hamilton functions can produce a model that DAGWorks would be able to save to mlflow. DAGWorks is more on the data transform side, and doesn't explicitly say how to represent a model.

deleted 4y ago
[deleted]

cheptsov 4y ago

Hey, the founder of dstack here. If I put it shortly, dstack allows you to define ML workflows as code and run them either locally or remotely (e.g. in a configured cloud). ML workflows here mean anything that you may want to do when you're developing a model - prepping data, training or finetuning a model, etc. The value - basically it automates running your workflows, without being dependant on any particular vendor. At the same time, you don't have to rewrite your Python scripts to use a particular API (because dstack is using YAML). We aim to build the most easy tool to run ML workflows - without making you use the UI of any vendor, or hustling with Kubernetes, custom Docker images, etc. MLFlow doesn't do what dstack does (automatic infrastructure provisioning) - unless you use Databricks.

krawczstef 4y ago
Nice! We had a similar abstraction at Stitch Fix.