5 ms·
Very cool! I'm interested in learning more about the memory model of multi-stage streaming systems. From my limited understanding of distributed data systems, w
by yingw787 7y ago
Very cool! I'm interested in learning more about the memory model of multi-stage streaming systems. From my limited understanding of distributed data systems, we went from saving files and checkpointing completely on disk a la MapReduce to a disk/memory hybrid with Spark RDDs and lazy job execution (not sure at all?) to pure streaming like Kafka. Could somebody please enlighten me? I'd love to learn more.
- tarun_anand 7y agoI would add that Kafka is a distributed commit log. Streaming is one of the applications. Spark also supports streaming. One should distinguish between the compute model like MapReduce, RDD etc and Data Model like streaming. HTH
- sanderjd 7y agoI just started reading Streaming Systems: https://www.goodreads.com/book/show/34431414-streaming-systems https://www.goodreads.com/book/show/34431414-streaming-syste.... Seems really great so far.