3 ms·
Trivial models can be rather common, in some application. For example Anomaly Detection per user or per sensor. The simplest models might basically be computing
by jononor 6y ago
Trivial models can be rather common, in some application. For example Anomaly Detection per user or per sensor. The simplest models might basically be computing the number of standard deviations* away the new datapoint is from a typical distribution, and then having a well-chosen threshold on this. Since these models are per entity, if one has 100'000 sensors deployed, then one gets 100'000 such trivial models.
Inference might run every 1 hour or so, and consume last 48 hours of data for example. Training might run once per week on the last 4 week.
This is to me a good candidate for an ML workload to run directly in a database:
- Low compute vs storage ratio for the models.
- High number of models.
- Often only want a small subset of data as input to model (a few numbers typically).
- Relational data highly relevant, for contextual data around the entity.
- Simple models with few parameters to store.
- Frequent updates to models.
- Historical models interesting. To implement checking new models against old, running in parallel
Other examples with similar characteristics would be Timeseries Forecasting on many different time-series. Could be sensor data, stock tickers or whatever.
* Better to use a robust analog such as Median Absolute Distance.
- jononor 6y agoBasic Anomaly Detection can be done with no support from the database (example: https://towardsdatascience.com/anomaly-detection-with-sql-7700c7516d1d https://towardsdatascience.com/anomaly-detection-with-sql-77...). But more effective (but still rather simple) models, like using Malahobis Distance, or kNearestNeighbours distance is harder to do efficiently. One rather mature project for doing ML integrated in SQL databases is Apache Madlib. It has been around since 2015. https://madlib.apache.org/ https://madlib.apache.org/
- TuringNYC 6y agoOn more thing: easy ingress-egreess for DB <---> Model w/o resorting to networks, encodings, decodings, etc.