3 ms·
Spark is specifically useful for querying streaming data. How would a distributed database help with that? You'd have to build your own stream executor on top o
by aloknnikhil 7y ago
Spark is specifically useful for querying streaming data. How would a distributed database help with that? You'd have to build your own stream executor on top of that.
- dominotw 7y agoisn't all data streaming data? whats so specail about streaming data.
- truth_seeker 7y agoAgreed. at the same time, Building stream execution pipeline is not rocket science. I am not saying modern distributed SQL Databases are exact replacement or clones of Spark. I am saying with little more help from the application server they are much more capable than Spark. You can use the following options individually or in combination. Option 1 : Pipeline DB extension (PostgreSQL) Option 2 : Service broker in commercial SQL databases or building PUSH/PULL queue if not supported. There are many libraries in each programming language which tries to do that. Also see option 4. Option 3 : Using CDC or Replication for synchronous or asynchronous streamed computation on single or multi node cluster Option 4 : Transducers. For example, you can compose many sql functions or procedures to act on a single chunk of data instead of always doing async streamed computation after each stage of transformation.