4 ms·
Consider this scenario. You can buy two different databases, one using your ML learned indexes, or another using the hash discussed in the article. For the has
by tensor 9y ago
Consider this scenario. You can buy two different databases, one using your ML learned indexes, or another using the hash discussed in the article.
For the hash database, it's 2x faster, up to 20x more memory efficient, and doesn't need training. It works on your data reliably from day you spin it up and throughout any changes to the stored data.
Meanwhile, your ML indexed database is going to be much slower out of the box, it will only be fast after learning on the data. Worse, if the data changes significantly it will need need to be trained again before being performant. And after all this, it still won't be as fast as the hash database.
For development, the ML based index technology won't be faster to build either. If anything, it will probably be much slower as there are all sorts of tricky nuances to building production grade ML systems. When do you train? How do you validate the training data? What do you do if the results don't validate to a sufficient accuracy?
I think it's fascinating that ML can be applied to problems like indexing, but if you can produce a deterministic theoretically optimal algorithm, you should use that instead of ML every time.