3 ms·

It's rare that I step into a conversation in software fields where I am completely at a loss for what is being discussed. This is one of those conversations. I

It's rare that I step into a conversation in software fields where I am completely at a loss for what is being discussed. This is one of those conversations. I have no idea what the description of the project means, and after reading the conversation here at HN, including people quoting references and such that explain what it does, I still don't really know what it does. I'm not complaining about not understanding, I just had one of those moments of "wow, the world of software is really big and there are vast, heavily funded, corners of it that I've never even heard of".

luckydata 11y ago
The short version: any complex data processing done at scale will have a lot of steps and fail a lot. To avoid having to constantly monitor everything, you have systems called "workflow managers" that help you make sense of those multi steps monsters and manage their dependencies, their failure etc... If you never had to process that kind of data (we're talking about Google, Facebook and Twitter kind of data) then you probably have no exposure to this kind of system.