I'd love to hear more what you have in mind! Feel free to drop by at our chat at https://gitter.im/metaflow_org/community https://gitter.im/metaflow_org/communi

I'd love to hear more what you have in mind! Feel free to drop by at our chat at https://gitter.im/metaflow_org/community https://gitter.im/metaflow_org/community While it is true that auto-sizing resources is hard and the easiest approach is to oversize @resources, the situation isn't as bad as it sounds: 1) In Metaflow, @resource requests are specific to a function/step, so you end up using resources only for a short while typically. It would be expensive to keep big boxes idling 24/7 but that's not necessary. 2) You can use spot instances to lower costs, sometimes dramatically. 3) It is pretty easy to see the actual resource consumption on any monitoring system, e.g. CloudWatch, so you can adjust manually if needed. 4) A core value proposition of Metaflow is to make both prototyping and production easy. While optimizing resource consumption may be important for large-scale production workloads, it is rarely the first concern when prototyping. In practice at Netflix, we start with overprovisioning and then focus on optimizing only the workflows that mature to serious production and end up being too expensive if left unoptimized. It turns out that this is a small % of all workflows.