4 ms·

> have a database that knows about all queries in your app, and optimizes collectively I’ve had this idea floating around in my head for a while: if you could

by klysm 4y ago

> have a database that knows about all queries in your app, and optimizes collectively I’ve had this idea floating around in my head for a while: if you could take all the queries and the schema, you should be able to compile that to a specialized binary that only does those queries. I have no idea how to accomplish that but I bet you could do a lot of nifty optimization. Migrations get hard but I wonder how good it would be

vaughan 4y ago
Makes me think of the current frontend trend of compiled reactive ui frameworks like Svelte. Also the SQLite bytecode engine. https://www.sqlite.org/opcode.html https://www.sqlite.org/opcode.html
- klysm 4y ago
  Perhaps a JIT for that bytecode isn't really that different
jandrewrogers 4y ago
This has been prototyped several times over the decades. Migration is one significant limitation as you note, but the other is that the metaprogramming required to code gen a highly optimized database for a specific data model and workload is insanely complex, and someone has to write and maintain that with the very low defect rate people expect of databases. The performance is excellent but modern databases tend to be bandwidth-bound for many workloads anyway, so the benefit will be significantly limited.
- klysm 4y ago
  Another limitation is debugability. I frequently run all kinds of queries against databases to figure out what's going on and compiling in the set of queries in advance could significantly limit your visibility into the data.
  - convolvatron 4y ago
    its only a performance problem. there wouldn't be any reason to forbid arbitrary queries - they just might not have pre-built indices that make them go as fast as they possibly could
    - klysm 4y ago
      > there wouldn't be any reason to forbid arbitrary queries There is an argument for this, it's just probably not worth the tradeoff. The security benefit seems somewhat reasonable. Feels like the kind of tradeoff as a unikernel.
gavinray 4y ago
This is called DBToaster https://dbtoaster.github.io/ https://dbtoaster.github.io/ *"DBToaster: Higher-order Delta Processing for Dynamic, Frequently Fresh Views"∗ http://vldb.org/pvldb/vol5/p968_yanifahmad_vldb2012.pdf http://vldb.org/pvldb/vol5/p968_yanifahmad_vldb2012.pdf
- klysm 4y ago
  Thanks for the link! This looks super cool edit: so this only does the read side right? It doesn't look like this supports mutations

by klysm 4y ago

vaughan 4y ago

Makes me think of the current frontend trend of compiled reactive ui frameworks like Svelte. Also the SQLite bytecode engine. https://www.sqlite.org/opcode.html https://www.sqlite.org/opcode.html

klysm 4y ago
Perhaps a JIT for that bytecode isn't really that different

jandrewrogers 4y ago

This has been prototyped several times over the decades. Migration is one significant limitation as you note, but the other is that the metaprogramming required to code gen a highly optimized database for a specific data model and workload is insanely complex, and someone has to write and maintain that with the very low defect rate people expect of databases. The performance is excellent but modern databases tend to be bandwidth-bound for many workloads anyway, so the benefit will be significantly limited.

klysm 4y ago
Another limitation is debugability. I frequently run all kinds of queries against databases to figure out what's going on and compiling in the set of queries in advance could significantly limit your visibility into the data.
- convolvatron 4y ago
  its only a performance problem. there wouldn't be any reason to forbid arbitrary queries - they just might not have pre-built indices that make them go as fast as they possibly could
  - klysm 4y ago
    > there wouldn't be any reason to forbid arbitrary queries There is an argument for this, it's just probably not worth the tradeoff. The security benefit seems somewhat reasonable. Feels like the kind of tradeoff as a unikernel.

gavinray 4y ago

This is called DBToaster https://dbtoaster.github.io/ https://dbtoaster.github.io/ *"DBToaster: Higher-order Delta Processing for Dynamic, Frequently Fresh Views"∗ http://vldb.org/pvldb/vol5/p968_yanifahmad_vldb2012.pdf http://vldb.org/pvldb/vol5/p968_yanifahmad_vldb2012.pdf

klysm 4y ago
Thanks for the link! This looks super cool edit: so this only does the read side right? It doesn't look like this supports mutations