I run Shipyard (www.shipyardapp.com) a workflow orchestration platform that's designed to handle the scale of processing and moving large data sets. It sounds l

I run Shipyard (www.shipyardapp.com) a workflow orchestration platform that's designed to handle the scale of processing and moving large data sets. It sounds like we'd easily fit the bill for what you're looking for. We're not a workflow as code framework like some of those that you mentioned. We let you automate your existing code (upload, copy/paste, or sync with Github) in the cloud with built-in monitoring and alerts. Install any packages at runtime, run each script in a containerized fashion, and connect them all together with complex pathing. No code changes or infrastructure management required. If you're interested, feel free to reach me at blake[at]shipyardapp.com. Would love to help you tackle your use case!