3 ms·
Has anyone used Unison instead of Spark? Seems like a natural fit to me.
by ildjarn 3y ago
Has anyone used Unison instead of Spark? Seems like a natural fit to me.
- rlmark 3y agoWe wrote an article speaking to this use case here: https://www.unison-lang.org/articles/distributed-datasets/ https://www.unison-lang.org/articles/distributed-datasets/ It's a bit of a deep dive into some of the building blocks of the Remote ecosystem - so it talks quite a bit about how you'd implement something like Spark itself in Unison, but you can see how running data aggregations would work on the Cloud.
- stewoconnor 3y agoWe recently did a series of blog posts exploring how our remote programming model makes us a good fit for writing distributed map - reduce like programs. https://www.unison-lang.org/articles/distributed-datasets/ https://www.unison-lang.org/articles/distributed-datasets/ One of the real strengths of individual parts of your program being content addressed, and our ability system that lets us track side effects, we can have a programming model where you only need to talk abstractly about where your data is, and how you'd like to operate on it, and then we have the ability to have our cluster gossip about parts of your program that need to be shipped to where parts of your data is. One node can ask another node "please apply this function to the data you have" and the other node can gossip to get any missing definitions it needs for that function. You never have to talk about serialization or network connections or software distribution, we know how to move data for you, move code for you, and in some cases even cash partial program results.