The short version: any complex data processing done at scale will have a lot of steps and fail a lot. To avoid having to constantly monitor everything, you have

The short version: any complex data processing done at scale will have a lot of steps and fail a lot. To avoid having to constantly monitor everything, you have systems called "workflow managers" that help you make sense of those multi steps monsters and manage their dependencies, their failure etc... If you never had to process that kind of data (we're talking about Google, Facebook and Twitter kind of data) then you probably have no exposure to this kind of system.