3 ms·
> bespoke Python apps that run on some kind of cloud container host like ECS. For this, you could use for example https://flyte.org/ https://flyte.org/, if you
by sampo 4y ago
> bespoke Python apps that run on some kind of cloud container host like ECS.
For this, you could use for example https://flyte.org/ https://flyte.org/, if you have platform engineers that could set it up for you. But I don't think their website makes a good job of explaining what it is and what it does.
- krawczstef 4y agoyep some of these ML platform tools only make sense once you reach a certain scale/set of problems. For flyte, it was created because Lyft had a very heterogeneous set of tools one could build a ML pipeline with. So they made it really easy to integrate and importantly serialize, data between systems. E.g. sql -> python -> spark -> python. But if all your data fits in memory, or you only one system, you might not be a great fit for Flyte. What I try to do, is understand who created the platform, and understand the environment of that company. Then that will give you a better idea of whether it makes sense for you or not.
- kumare3 4y agoI do agree it makes sense fornscale, but if your data fits in memory Flyte native constructs shine. For example it will ensure your data is stored / serialized correctly. Allows you to use polars, vaex, duckdb etc. tbh I am a huge proponent of vertical scaling till you can get the mileage. It also supports - gpu allocation, spot instances and collaboration across multiple users. I do not think it is a wrong choice if you feel your complexity will grow. PS. I am a maintainer of the project
- krawczstef 4y agoYep - it's on my TODO to get https://github.com/flyteorg/flyte/issues/2627 https://github.com/flyteorg/flyte/issues/2627 over the line!
- kumare3 4y agoThank you for sharing. If you do not have platform engineers look at union.ai. They offer a manager version of Flyte