4 ms·
yep some of these ML platform tools only make sense once you reach a certain scale/set of problems. For flyte, it was created because Lyft had a very heterogene
by krawczstef 4y ago
yep some of these ML platform tools only make sense once you reach a certain scale/set of problems. For flyte, it was created because Lyft had a very heterogeneous set of tools one could build a ML pipeline with. So they made it really easy to integrate and importantly serialize, data between systems. E.g. sql -> python -> spark -> python. But if all your data fits in memory, or you only one system, you might not be a great fit for Flyte.
What I try to do, is understand who created the platform, and understand the environment of that company. Then that will give you a better idea of whether it makes sense for you or not.
- kumare3 4y agoI do agree it makes sense fornscale, but if your data fits in memory Flyte native constructs shine. For example it will ensure your data is stored / serialized correctly. Allows you to use polars, vaex, duckdb etc. tbh I am a huge proponent of vertical scaling till you can get the mileage. It also supports - gpu allocation, spot instances and collaboration across multiple users. I do not think it is a wrong choice if you feel your complexity will grow. PS. I am a maintainer of the project
- krawczstef 4y agoYep - it's on my TODO to get https://github.com/flyteorg/flyte/issues/2627 https://github.com/flyteorg/flyte/issues/2627 over the line!