4 ms·
Does anyone have any ML Packages that directly integrate with SQL Databases? I feel like the ETL market is well covered with a myriad of tools, but our data sci
by exabrial 7y ago
Does anyone have any ML Packages that directly integrate with SQL Databases? I feel like the ETL market is well covered with a myriad of tools, but our data scientists at the end of the day still want to extract giant CSVs b/c their python progs "just don't work natively" on an RDBMS (they probably do, but it's not the way it's done, and these guys aren't programmers for their first job anyway).
- olooney 7y agoThey exist, they're just not really state-of-the-art. Apache MADLib[1] for PostgreSQL and Greenplum. Everything is done inside of SQL; for example, to fit a linear regression model, you execute the query "SELECT madlib.linregr_train(...)". MLlib[2] for Spark now has a DataFrame based API (spark.ml) for easy integration with Spark SQL. Every such library I've seen has been a little behind the times, usually offering nothing more advanced than logistic regression, random forest, maybe a multi-layer neural network. I've also never seen a SQL-based ML library that offered GPU training. For reference, state-of-the-art neural network libraries like TensorFlow or PyTorch support GPU-based training (something like an order of magnitude improvment in training performance) and automatic differentiation (allowing for easy specification of complex models.) For trees, state-of-the-art now includes boosted tree algorithms like XGBoost or CatBoost. Random forest is still a workhorse, but a library which only provides random forest (like the two mentioned above) are a little behind the curve. It seems that libraries that focus on doing one thing well (ML) are able to offer more features and have an easier time keeping up with times than libraries that also take on the burden of SQL integration. Which isn't to say MADlib and Spark MLlib are bad - not everyone needs state-of-the-art algorithms all the time, and it can be convenient to be able to do so simple things fully inside the SQL environment - but they're never going to be cutting edge, so they're not going to get much attention. [1] https://madlib.apache.org/index.html https://madlib.apache.org/index.html [2] http://spark.apache.org/docs/latest/ml-guide.html http://spark.apache.org/docs/latest/ml-guide.html
- sixdimensional 7y agoSQL Server has had standard ML functions in SSAS for years “SQL Server Data Mining” extensions. In fact, I recall it may have been the first relational database platform to actually do this. It is obviously a commercial DB though.
- mmeasic 7y agoI believe that more and more providers will offer ML inside the database. You can check what Oracle is doing. Aside from that, some of the simple models can be fitted using SQL.