2 ms·
Agreed. at the same time, Building stream execution pipeline is not rocket science. I am not saying modern distributed SQL Databases are exact replacement or cl
by truth_seeker 7y ago
Agreed. at the same time, Building stream execution pipeline is not rocket science. I am not saying modern distributed SQL Databases are exact replacement or clones of Spark. I am saying with little more help from the application server they are much more capable than Spark.
You can use the following options individually or in combination.
Option 1 : Pipeline DB extension (PostgreSQL)
Option 2 : Service broker in commercial SQL databases or building PUSH/PULL queue if not supported. There are many libraries in each programming language which tries to do that. Also see option 4.
Option 3 : Using CDC or Replication for synchronous or asynchronous streamed computation on single or multi node cluster
Option 4 : Transducers. For example, you can compose many sql functions or procedures to act on a single chunk of data instead of always doing async streamed computation after each stage of transformation.