3 ms·
How much of this is still relevant with the upcoming 2.0 structured streams?
by ddispaltro 10y ago
How much of this is still relevant with the upcoming 2.0 structured streams?
- jagsr123 10y agoStructured streaming really makes spark much more competitive for streaming use cases by allowing streams to be seen as DataFrames and hence make it super simple to run continuous queries using SQL on streams, but, often you also need to work with historical data in your analytic query, continuously mutate data (which may require transactional semantics), make sure the data itself is HA (not just fault tolerant; important for low latency apps), etc. What we do is fuse in modern in-memory DB with spark (i.e. DB cluster nodes run spark executors) so there is no need to couple spark with some other data management cluster. That said, some of the APIs we introduced for SQL stream processing will be replaced by the new Structured streaming APIs.
- plamb 10y agoThere was a talk sort of on this topic at Spark Summit in June... Here is the part where 2.0 is mentioned: https://youtu.be/PViLT2E2WPQ?t=407 https://youtu.be/PViLT2E2WPQ?t=407