Oh and one thing I forgot to address above: > In my experience, you want to decouple the task-level orchestration mechanism from the nature of the data itself.

Oh and one thing I forgot to address above: > In my experience, you want to decouple the task-level orchestration mechanism from the nature of the data itself. No, that's a fantasy. The data is EVERYTHING. There is no abstract, platonic solution that works equally well with 100 rows or 100 billion rows. One of the unique things about Data Engineering that sets it apart from most other specialties is that we deal in bulk. The code we write takes significant wall-clock time to execute. Performance ALWAYS matters. Never forget that!