3 ms·
Please give me an example. I can't think of any transform which cannot be done by using SQL or inbuilt Functions or new UDF.
by truth_seeker 7y ago
Please give me an example. I can't think of any transform which cannot be done by using SQL or inbuilt Functions or new UDF.
- ai_ja_nai 7y agoSpark != SQL It's also graph analysis and ML models.
- truth_seeker 7y agoGraph analysis -> Recursive common table expression (https://www.postgresql.org/docs/current/queries-with.html https://www.postgresql.org/docs/current/queries-with.html) ML models - I already mentioned how to uplift R and Python functions to SQL function. even if you are not using PostgreSQL many other databases help you with uplifting and interfacing with existing ML libraries through FFI
- brokensegue 7y agotrain a set of sklearn models one each per a random partition of the data (computed distributed). then combine all those models using averaging and evaluate them all against an even larger dataset. how do you do that in SQL
- truth_seeker 7y agoSharding the table can help scale the problem across many machines and as I mentioned earlier you can use PL/R or PL/Python language extension to lift all sorts of ML functions to SQL functions.
- brokensegue 7y agoI'm unfamiliar with PL/Python. Can you have a Python object be the returned value of a sql query? Because that's a requirement of my example.
- deleted 7y ago[deleted]
- nxpnsv 7y agoIt’s also possible to do a lot in excel, it is just not always the best tool for the job.