4 ms·
The idea of doing predictive ML inside of, say, MySQL, sounds tremendously appealing. E.g. if you could add something like a "predictive index" that would model
by crazygringo 6y ago
The idea of doing predictive ML inside of, say, MySQL, sounds tremendously appealing. E.g. if you could add something like a "predictive index" that would model one column based on certain other columns, and with a variety of "prediction types" (just like choosing index types of b-tree, hash, etc.).
Although instead of a SELECT statement on existent rows, you'd need to use something like a PREDICT statement on non-existent rows:
PREDICT product_category
FROM invoice_data
WHERE item_description = "Packaging design"
AND vendor_code = "VENDOR-1676"
The big question for me, however, would be: when would the predictive model be updated? It seems too computationally expensive to update it on every UPDATE, INSERT and DELETE. I mean, if it's of any decent complexity, it would require a full table read every time it was updated, no? Would you have to manually issue a command to update it? How long would that take?
But it absolutely seems like a very natural place to predictive capabilities -- directly into the database.
- jimsparkman 6y agoSomething roughly similar is possible in the AWS “big data” stack. The Athena SQL engine can reference Sagemaker models for inference. This is effectively exposed via UDFs. I haven’t used it yet personally but it looks super useful.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- zitterbewegung 6y agoThe power of SQL is that you don't even have to make the decision "when would the predictive model be updated? " . This could be configurable and be dependent on the data that you can find in the tables. Also, another solution that is sort of the opposite is Datasette. https://datasette.readthedocs.io/en/stable/ https://datasette.readthedocs.io/en/stable/
- jsemrau 6y agoThat sounds more like fuzzy inference rather than prediction.
- cf 6y agoWhy not? If you pick your model well incremental updates to the model can be made fairly cheap. An open source example that's relatively usable can be found at http://probcomp.csail.mit.edu/software/bayesdb/ http://probcomp.csail.mit.edu/software/bayesdb/
- deleted 6y ago[deleted]
- zelly 6y agoDeciding how to schedule the model training seems to be the easiest possible part of such a system.
- arauhala 6y agoIt does sound appealing, doesn't it :-) Regarding the model: as described in the article, you can create specialized ML models in millisecond scale. This means that all computational, latency and consistency issues disappear, as you don't need to update models. The not so nice side of this approach is that there are currently limitations on scaling, so that it may work fine with 100k samples, but becomes somewhat slower with 1M samples and likely won't work in 10M and 100M scale. Still, like described in the article, I believe this issue is solvable within near future, and it's not an issue for many/most applications.
- wodenokoto 6y agoGoogle Clouds big query has an option for defining ml models in sql. R can deploy several models as sql queries. MSSQL also has options for this.
- arauhala 6y agoThe big thing in Aito is still, that it is 'modelless'. You don't need a prepared model to do a prediction, because the model is done ad hoc / lazily for the query. It is a much easier workflow, because you don't need to care about the models and you can have very free format / templated queries like this: predictCaseFieldForCustomer(customer, department, case, predictedField) { return aito.query({ from: f'customer-$customer', where: { $on : [ { case : case }, { department : department } ] }, predict : predictedField }) }
- dvasdekis 6y agoYep, that's dependent and independent variables in a simple linear regression (SLR). Postgres has SLR built in. Madlib is an extension to Postgres that does much more.
- arauhala 6y agoPlease correct me if I'm mistake, but doesn't that require a separately prepared model in both Postgres and madlib? You cannot simply request for an arbitrary prediction in a single & simple query.