12 ms·
I enjoyed this explanation, thank you. Curious to know what the differences are vis-a-vis big data tools like Hadoop or Spark; as a user of those tools I recog
by karanke 8y ago
I enjoyed this explanation, thank you.
Curious to know what the differences are vis-a-vis big data tools like Hadoop or Spark; as a user of those tools I recognized a lot of common failure patterns in the example above. Thanks!
- _asummers 8y agoI'm unfamiliar with Hadoop/Spark beyond knowing what they are and vaguely where they'd be used, but I imagine they tend on the "well we can just recalculate sub-jobs" failure models, instead optimizing developer speed over computational speed. Though that is wild speculation. You wouldn't use Erlang/Elixir for doing calculation because that's just not what it's good at. But you might use it as something to manage jobs in a larger distributed system perhaps. Though my suspicion there is you may run into tooling impedance mismatches as you get deeper into the failure modes. Sorry I can't speak more knowledgeably about that. I'm curious if any WhatsApp folks are around and can comment on if they ever made their Erlang systems talk with their Hadoop/Spark stuff and what that all looked like.