3 ms·
What if you create something that is optimized for performance without offering the full SQL language? Just a container to store small files with padding to avo
by Tepix 7y ago
What if you create something that is optimized for performance without offering the full SQL language? Just a container to store small files with padding to avoid the expensive open/close calls.
- hizanberg 7y agoLook at Google’s LevelDB and Facebook’s RocksDB.
- pjc50 7y agoRead-only versions of this are extremely common for games; I first encountered it with 90s DOOM WAD files.
- typon 7y agoWrite API abstractions that offer baked and templated Sql queries for your specific use case. Flexible enough for future growth, easy enough for every day normal use without inventing a whole new system.
- iforgotpassword 7y agoMinecraft vastly improved performance when it switched from having one file per 16x16 map area to just stuffing 32x32 of those chunks into one file with a simple index at the beginning.
- liability 7y agoI'm not sure how much better you'd get than sqlite if you were using a straight forward rowid select with a prepared statement. Sqlite is pretty damn snappy for something like that and using a prepared statement amortizes the cost of compiling the SQL.
- zzzcpan 7y agoYou can get a point read with just a single aligned random I/O operation on disk with this. I don't see how a B-tree based storage can compete. And even more optimizations are possible, like grouping objects often accessed together into a single batch to retrieve the whole group in just one random operation too.
- liability 7y agoI'm sure you can go faster, but how much faster? The rowid b-tree lookup, with the b-tree in memory, will almost certainly be blazing fast compared to the actual disk read.
- hawski 7y agoAs sibling comments stated it's common in games. A well known open source VFS library is PhysicsFS¹, previously it was called PhysFS AFAIR. ¹ https://icculus.org/physfs/ https://icculus.org/physfs/