3 ms·
Yes, like these libraries. Think, for example, sharding, where you run lots of database instances per disk, this requires strictly bounded memory usage and no b
by zzzcpan 7y ago
Yes, like these libraries. Think, for example, sharding, where you run lots of database instances per disk, this requires strictly bounded memory usage and no background operations. Or running on a disk full of bad blocks with constant retries.
- ngrilly 7y agoBut SQLite provides an API to set memory usage limits [1] and doesn't use background operation as far as I know. I'm not sure about RocksDB (I think the "levels" compaction runs in the background by you can control it). [1] https://www.sqlite.org/malloc.html#_setting_memory_usage_limits https://www.sqlite.org/malloc.html#_setting_memory_usage_lim...
- zzzcpan 7y agoWell, it was just an example, it wouldn't really work either way, would be too slow for HDDs and would still require handling of all the disk reliability and performance issues. But if I were to do it today and store metadata in an SQLite database I would use a single database for all the shards and use its VFS API where I would add I/O scheduling, remapping of blocks with redundancy into different blocks and automatic recovery, marking of bad and slow blocks, and maybe even scrubbing. Looks like half of the storage engine already and it would still be somewhat slow.