3 ms·
Do this: http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/ http://blog.cloudera.com/blog/201
by nl 11y ago
Do this: http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/ http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-ti...
Then at the end instead of writing to HBase you can write JDBC to do the insert.
- sixdimensional 11y agoNice link. Spark jobs run Java code anyway, so why not write to JDBC with the results of processing. Makes perfect sense.
- threeseed 11y agoJDBC works great if you have a large RDD that you want to persist in one go. But if you are writing row by row you will need to implement your own batching algorithm and connection pooling to get any decent performance.
- gglanzani 11y agoYou could always use mapPartition to open one connection per partition.