I'm unfamiliar with Hadoop/Spark beyond knowing what they are and vaguely where they'd be used, but I imagine they tend on the "well we can just recalculate sub

I'm unfamiliar with Hadoop/Spark beyond knowing what they are and vaguely where they'd be used, but I imagine they tend on the "well we can just recalculate sub-jobs" failure models, instead optimizing developer speed over computational speed. Though that is wild speculation. You wouldn't use Erlang/Elixir for doing calculation because that's just not what it's good at. But you might use it as something to manage jobs in a larger distributed system perhaps. Though my suspicion there is you may run into tooling impedance mismatches as you get deeper into the failure modes. Sorry I can't speak more knowledgeably about that. I'm curious if any WhatsApp folks are around and can comment on if they ever made their Erlang systems talk with their Hadoop/Spark stuff and what that all looked like.