3 ms·
Ask HN: Is there an equivalent of LLVM for databases?
Being very impressed with what LLVM did for compiler generation, I wondered if there is something similar for databases?
What I mean is a low-level system that would allow me to make use of some primitive facilities that databases are based on (indexes, heaps, storage engines, ...) and use them to build my own database/data processing systems. Is there such a thing?
I think this could be very interesting as I often find myself in a situation where I have a good idea of how to access data using a given set of indexes, but am not able to do this reliably because the query planner does give a different, less efficient plan. And please don't tell me that the query planner always knows better, as often it does not even provide a way to specify which goal I'm optimizing for (e.g. predictable performance / throughput vs. better but unpredictable performance).
Having access to such an open, optimized platform for database generation would make it much easier building systems that are highly optimized for a given use case instead of being designed for general-purpose data storage.
- brudgers 10y agoMy 10,000 foot high understanding is that the kind of performance you're seeking is achieved by the dark arts of database tuning. At a certain level of abstraction one might think about the query planner's default mode as context free and your goal as adding context sensitivity to it. At the other end of the abstraction tree, it might be that a relational database is the wrong tool or that a row-based relational database is less right than a column based database. If you can see how the data ought to be organized for performance, then it might make sense to build project specific data structures. Or use an in-memory database. There are lots of options besides relational databases. Good luck.
- ThePhysicist 10y agoYes I agree with what you say, sometimes it's just inconvenient to manipulate a query planner using a high-level language instead of using a more direct approach. Also, while many databases provide a way to work with underlying representations of their data, it is still quite difficult or impossible to build something entirely new (I.e. a new application) with them.
- brudgers 10y agoBecause I don't have details below the level of database and query, I had to imagine them up to write an answer. It seems like your use case is a bit unique.