3 ms·
They exist, they're just not really state-of-the-art. Apache MADLib[1] for PostgreSQL and Greenplum. Everything is done inside of SQL; for example, to fit a li
by olooney 7y ago
They exist, they're just not really state-of-the-art.
Apache MADLib[1] for PostgreSQL and Greenplum. Everything is done inside of SQL; for example, to fit a linear regression model, you execute the query "SELECT madlib.linregr_train(...)".
MLlib[2] for Spark now has a DataFrame based API (spark.ml) for easy integration with Spark SQL.
Every such library I've seen has been a little behind the times, usually offering nothing more advanced than logistic regression, random forest, maybe a multi-layer neural network. I've also never seen a SQL-based ML library that offered GPU training.
For reference, state-of-the-art neural network libraries like TensorFlow or PyTorch support GPU-based training (something like an order of magnitude improvment in training performance) and automatic differentiation (allowing for easy specification of complex models.) For trees, state-of-the-art now includes boosted tree algorithms like XGBoost or CatBoost. Random forest is still a workhorse, but a library which only provides random forest (like the two mentioned above) are a little behind the curve.
It seems that libraries that focus on doing one thing well (ML) are able to offer more features and have an easier time keeping up with times than libraries that also take on the burden of SQL integration. Which isn't to say MADlib and Spark MLlib are bad - not everyone needs state-of-the-art algorithms all the time, and it can be convenient to be able to do so simple things fully inside the SQL environment - but they're never going to be cutting edge, so they're not going to get much attention.
[1] https://madlib.apache.org/index.html https://madlib.apache.org/index.html
[2] http://spark.apache.org/docs/latest/ml-guide.html http://spark.apache.org/docs/latest/ml-guide.html